Google is part of a profit-maximizing company. Slowly, they are transitioning from showing relevant results and some ad in a separate box to results that maximize profit directly because they’re sponsored or whatever. The founders even explained it in one of their papers.

Speculation: Crawling the web is expensive. So it makes sense that they decide for each crawled page if it's profitable to put that into the index. In the long run they might not crawl the web at all. People will just pay them to get into the index.