Maybe the concept of "web search" needs to be reimagined all together?
Perhaps the idea of creating a master index of the content of multiple pages on nearly every site is the thing that is not sustainable. Instead, "search this site" needs to be made great again. Individual websites could manage their own search in a way that complies with a standard API that can be consumed by meta-search engines. Rather than indexing pages in the traditional way, meta-search engines instead use a heuristic or AI model to decide what sites are going to have the kind of information you are searching for, perform your query on theirs, and return the aggregated results to the user. The less the algorithm understands the significance or meaning of the query, the more generalized its approach can be. For instance, if it thinks that you're searching specifically for opinion-based content that will appear on blogs and forums, then it will target a federated search engine that indexes those things specifically. But if you are searching for information on making beer at home, it will know to target and weight the search engine on brewersfriend.com and homebrew.stackexchange.com. Although this sounds not that different to how search currently works today, remember that this idea is about having search become more federated and more standardized, and for meta-search to select federated indexes rather than own a god-index. A user of a meta-search can pick and choose what indexes they want available in their searches in case they find any of them to be either superior or particularly problematic, and the meta-search can optionally adjust its understanding of a particular user's queries.
The way I see it, traditional search will continue to decline in part because it's not sustainable, but also in response to AI allowing them to become "answer engines". Although a lot of people do want an answer engine, this isn't for everyone. I think there will always be a market for people looking for content on specific webpages. Whatever that thing is that someday snatches that market away from The Google, if it's going to be successful, won't survive on the current concept of what "search" is.
Perhaps the idea of creating a master index of the content of multiple pages on nearly every site is the thing that is not sustainable. Instead, "search this site" needs to be made great again. Individual websites could manage their own search in a way that complies with a standard API that can be consumed by meta-search engines. Rather than indexing pages in the traditional way, meta-search engines instead use a heuristic or AI model to decide what sites are going to have the kind of information you are searching for, perform your query on theirs, and return the aggregated results to the user. The less the algorithm understands the significance or meaning of the query, the more generalized its approach can be. For instance, if it thinks that you're searching specifically for opinion-based content that will appear on blogs and forums, then it will target a federated search engine that indexes those things specifically. But if you are searching for information on making beer at home, it will know to target and weight the search engine on brewersfriend.com and homebrew.stackexchange.com. Although this sounds not that different to how search currently works today, remember that this idea is about having search become more federated and more standardized, and for meta-search to select federated indexes rather than own a god-index. A user of a meta-search can pick and choose what indexes they want available in their searches in case they find any of them to be either superior or particularly problematic, and the meta-search can optionally adjust its understanding of a particular user's queries.
The way I see it, traditional search will continue to decline in part because it's not sustainable, but also in response to AI allowing them to become "answer engines". Although a lot of people do want an answer engine, this isn't for everyone. I think there will always be a market for people looking for content on specific webpages. Whatever that thing is that someday snatches that market away from The Google, if it's going to be successful, won't survive on the current concept of what "search" is.