Vlad from Kagi here, thanks for posting. The RSS feed unexpectedly broke (edit: the feed is back up! [3]) just as we published the blog post, in the true spirt of "small web" :) Should be up in 30 minutes which will enable the site to function too (it uses the same feed).
This has been a personal pet project of mine and I spent considerable time getting my hands dirty with the code, as the team was busy with other initiatives. When I said the "feed broke" for the launch I meant I broke it. Software is messy especially for an old school dev. I learned in the process I am not a very good coder anymore (if I ever was one?), constantly going back and fixing stuff I previously thought was solid. Check it out in the linked repo [1].
Most importantly - I found the site replace the need for discovery for me, and getting to know various different humans and their writing felt good! A lot of unexpected stuff surfaced and the web felt close again. I think there is a glimpse of hope in the concept and I hope you see it too. And the improvements to search quality and diversity this brings are real.
You can check the list of included websites here [2]. And all the recent posts already surface in Kagi results (for relevant queries).
I really enjoy that this feature got released. Adding more visibility to the indie/small web is a great thing. I've complained in the past on HN that any useful personal blog is buried under spammy content in search engines.
It would be nice to also be able to just search within the small web, maybe using a lens in the future?
Regarding the topic of self promotion, I would disagree with the current rules and I would ask you to allow people to self promote. As long they have an old enough blog, maybe even cut that down to a year, would be helpful. Most users on the small/indie web lack visibility and this would do them service. My blog is already within the index because I think it might have been picked up during the "HN share your blog" post that happened a while ago, but others might not have been that lucky.
There is already a lens called the non commercial web or something that I suspect overlaps to some degree,but I don't think they are equal and I don't know if I am right.
The more I uses Kagi (paid) the more I am feeling sad for the possibility that it might go away if it doesn't get more traction.
For me you represent an incredible accomplishment: the first search engine that gives better results than Google, respects privacy, offers customization and so much more.
Just to say: the search engine frog has long been boilt and it is amazing what crap people have to put up with when searching for anything. How jarring the experience of seeing TV commercials is after years of Kodi, is how the experience is of moving back to an ad-supported search engine after now months of Kagi.
Youtube doesn’t show ads in my country. Anytime I use a vpn or I travel, I can’t believe how bad the free tier is. My twitch consumption has gone way down because of the ads. Watching the last NBA season was so annoying. I felt more than half of the broadcast was ads.
Isn't this more likely to have been inspired by marginalia* than a personal blogs thread? Doesn't seem to have the same results from the Apple Watch example, but it's what immediately came to mind for me when I read this post's title.
Vlad was dabbling with similar things to Marginalia with Teclis around the same time Marginalia first made headlines.
And to be fair, I was simultaneously inspired by the blog thread to build a curated blog filter for my search engine, which led to a series of changes that overall tends to promote more of this types of results in general.
The fact that we're several to have similar ideas sort of validates the ideas I think.
I'm not complaining, glad there's more tools for searching outside the scope of SEO spam or content mills, and competition can only make the niche better (presumably, unless it gets big enough to incentivize disguising commercial content as niche blogs). Just felt like it might be a bit disingenuous to quote a blog list as inspiration for a search engine focused on "the non-commercial part of the web" which is exactly your engine's aim and wording, didn't know they've been working on small web initiatives since a similar timeline.
Hopefully you both find success and with it uplift hobbyist websites.
Why do you let YouTube channels have advertisements/sponsors but require sites to be ad-free?
I understand the spirit of it and don't have any counter examples but seems like a bummer if someone has a nice indie blog but can't be added because they have a few ads or a sponsored post.
Most of the focus was on the website curation. The YT channels were curated from the initial list of sources [1] and then automatically filtered by recency and subscriber count criteria.
That is to say that very little other effort was made to curate the YT channels and we expect the user community will contribute to edit the list.
Also I was under impression that all YouTube channels had ads, so that is why this was not considered as a seperate criterion.
Thanks for clarifying. Yeah we should establish a criterion around that. Curating YT is much more work it feels like. Let's see how "Small YouTube" [1] feels like to begin with and we can take it from there.
By the way, you can perhaps check https://nebula.tv for a list of good YouTube channels (not all of them are great IMO, but a lot of people who make content on Nebula fit the "small web" criteria)
Now that I think of it, perhaps you can call Nebula "the Kagi of video streaming". Maybe you guys could collaborate somehow in the future ;D
Vlad helped me troubleshoot why my Orion browser wasn't syncing bookmarks (turned out to be a config issue on my device) and we got the fix added into the official documentation.
Imagine getting that level of support for any other web browser!
I kinda wish there was a sort of proper blog-DMOZ. Would make this sort of stuff a lot easier. Right now everyone and their grandma is curating their own blog list. Lots of wasted effort that could be saved by collaborating somehow.
I agree but it’s incredibly hard to collaborate on this kind of projects because everyone has their own take on what should and shouldn’t be included.
Curation is still hard. Also, I personally don’t mind having different curated directories because those will reflect the taste and interests of the curator.
I figure collaborate on answering the easy questions, is the website online, is it chock full of spam, is it a blog about horses or about keyboards, etc. Some of that could be automated (like onlineness, large change detection, etc.), but some of it needs manual supervision.
Right now everyone who is running any sort of curated discovery service needs to answer these same questions about roughly the same site.
ooh.directory gained some traction a few month ago [0] - and they are steadily adding blogs although they do it slowly. Manually reviewing and trying to have some diversity is no easy task.
Because curation to keep spam out would be work. And how do you prevent trojan horses, where an initially promising blog is turned over to SEO once included in the official dmoz?
There’s also the issue of quality. Do you just include everything that is a blog, no matter how low quality it is?
And if the answer to that is no, then where do you draw the line when it comes to quality?
Curating anything is complicated and different people will have different opinions which is why you end up with different lists curated by different groups of people with different ideas about what should and shouldn’t be included.
Semi-unrelated, but is there any reason you don't make your search API as turn key as the other APIs?
It's expensive enough that I can't imagine anyone repackaging it profitably (2x Bing search prices for me) but having to email someone adds just enough friction to discourage a lot of tinkerers from even trying it.
I have been developing a "small web" static site, forked and customized a templating engine into a static site generator just for my site. Even wrote a new post recently, last week or so. How can I add it to the list? The GitHub says to make a pull request, is that really all there is to it?
Edit: just saw this:
> Do not submit your own website.
I see. I'm okay with that. Maybe it will show up there one day.
Are English "Small Web" results included even if I use the "Sweden" search region?
I noticed the example result for useyourloaf wasn't included if I switched it to "Sweden" and not sure if this is just an oddity or if the entire feature is nerfed because I just leave my locale on all the time.
This has been a personal pet project of mine and I spent considerable time getting my hands dirty with the code, as the team was busy with other initiatives. When I said the "feed broke" for the launch I meant I broke it. Software is messy especially for an old school dev. I learned in the process I am not a very good coder anymore (if I ever was one?), constantly going back and fixing stuff I previously thought was solid. Check it out in the linked repo [1].
Most importantly - I found the site replace the need for discovery for me, and getting to know various different humans and their writing felt good! A lot of unexpected stuff surfaced and the web felt close again. I think there is a glimpse of hope in the concept and I hope you see it too. And the improvements to search quality and diversity this brings are real.
You can check the list of included websites here [2]. And all the recent posts already surface in Kagi results (for relevant queries).
[1] https://github.com/kagisearch/smallweb
[2] https://github.com/kagisearch/smallweb/blob/main/smallyt.txt
[3] https://kagi.com/smallweb