Four year retrospective
Today is the 4 year anniversary of searchmysite.net (the open source search engine for the indieweb / small web / digital gardens, which heavily downranks pages with adverts, and aims to pay running costs via a listing fee and search-as-a-service), so that seems a good point for a quick update on its current status.
Highlights from the past year
The biggest and most exciting news from the past year was of course the redesign and first major open source contributor in Aug 2023. More in the dedicated blog post.
The vector search and chat-with-your-website functionality (mentioned in the Three year retrospective) has also been built, the servers upgraded to cope with the self-hosted LLM (served via TorchServe) and the extra load of generating embeddings during indexing, and the code deployed to production. However, I’ve not put live links to the new interface because the results aren’t especially good yet to be honest. I don’t think retreival augmented generation is quite the silver bullet some people have suggested it is, and suspect the best results will need some kind of hybrid search. Could be a good project for another contributor to pick up though.
Usage levels
Usage levels have been improving. For the first 3 years it was generally hovering around 10 real users a day (see e.g. Progress update Q1 & Q2 2021), excluding the occasional spikes, and of course excluding the massive amount of SEO spam bot activity. However, since the spike in Jan 2024 it has been around 20 a day, and since the spike in Mar 2024 is has been around 30 a day:
Not entirely sure why to be honest. Perhaps it is part of a growing disillusionment with all the AI generated slop which is starting to litter the internet, or a reaction to the falling quality of results from the major search engines.
Anyway, the true indicator of searchmysite.net’s success (or otherwise) would be if people want to come back periodically (rather than visit once and feel they’d “been there done that” and never come back), but I’m not sure the current stats can really determine if that is happening or not.
Running costs vs paid listings
Running costs have increased quite a bit following the server upgrades for vector search, and the number of subscriptions has fallen slightly since the previous year, but it still doesn’t look too far off break even point:
| Year | Expenses | Income |
|---|---|---|
| Jul 2020 – Jun 2021 | £351.57 | £57.25 |
| Jul 2021 – Jun 2022 | £456.04 | £80.51 |
| Jul 2022 – Jun 2023 | £94.51 | £137.32 |
| Jul 2023 – Jun 2024 | £174.79 | £125.06 |
| Totals: | £1,076.91 | £400.68 |
Given the time and effort required for support and maintenance isn’t enormous (although it is probably way more than people realise - maybe it would be worth a dedicated blog post about that), I think it should be fine to keep searchmysite.net going for the time being.
Closing thoughts
It has been interesting to see that some people instantly get what searchmysite.net is about, while others don’t, and sometimes the people you’d expect to get it don’t, and vice versa. As a recap, searchmysite.net is not meant to replace the major search engines or be a general purpose search that anyone would use on a daily basis - it is for “web surfing”, as in that leisure activity people used to perform when they were a little bored, or periodically searching for things relating to hobbies or interests. I don’t know if a new interface would help make the objective more apparent or not.
But the subsequent appearance of other search engines with similar objectives does suggest there is an interest in finding real content from real people. There was even a new service announced in Feb 2024 which provides a search as a service for personal sites, in addition to downranking sites with adverts on them, so again that suggests the original ideas behind searchmysite.net were sound.