Almost all searches on my independent search engine are now from SEO spam bots

Introduction

searchmysite.net was launched nearly 2 years ago to help people discover all the great original content on personal and independent websites which is so hard to find now that the major search engines have become swamped by SEO spam. It employed a number of novel techniques to try to avoid the same fate, such as having a community-driven curation and moderation layer, heavily downranking pages containing adverts to reduce the incentive for SEO spam, and aiming to pay the running costs from its search as a service rather than by selling user data to advertisers.

That makes it especially tragic to report that nearly all the traffic to the site is now from SEO spam bots, presumably searching for all that elusive SEO spam-free content. Here’s the stats for recent requests to the search page¹:

There are two parts to this problem:

1. Usage by spam bots has increased dramatically.
2. Usage by real users is almost non-existent.

Usage by spam bots has increased dramatically

I’ve always had some activity from bots², but it has been manageable. However, in mid-April 2022, bot activity started to increase dramatically.

I didn’t notice at first because the web analytics only shows real users, and the unusual activity could only be seen by looking at the server logs. I initially suspected that it was another search engine scraping results and showing them on their results page, because the IP addresses, user agents and search queries were all different. I then started to wonder if it was a DDoS attack, as the scale of the problem and the impact it was having on the servers (and therefore running costs) started to become apparent.

After some deeper investigation, I noticed that most of the search queries followed a similar pattern, e.g. containing “Powered by” and the name of some blog or forum tool and then the search query. It turns out that these search patterns are “scraping footprints”. These are used by the SEO practitioners, when combined with their search terms, to search for URLs to target, implying that searchmysite.net has been listed as a search engine in one or more SEO tools like ScrapeBox, GSA SEO or SEnuke.

It is hard to imagine any legitimate white-hat SEO techniques requiring these search results, so I would have to imagine it is for black-hat SEO operations, e.g. looking for quality links to which to attempt to automatically post backlinks, or for original content to copy to link farms, or for good sources of email addresses, etc.

I sincerely hope that none of the sites in searchmysite.net have been adversely affected by this. I have been doing my best to block the SEO spammer requests since discovering the issue³, but I am concerned it could be a losing battle, and one with nothing to gain by winning.

Usage by real users is almost non-existent

Most of the tiny number of real users have come from links posted to places like Hacker News, and there is almost no organic traffic from other search engines. I wrote about this in Progress update Q1 & Q2 2021, including stats on the top traffic sources, along with details of all the white-hat SEO operations I had performed to try to improve ranking within the search engines. If you were into conspiracy theories you could claim that the major search engines were trying to stifle the competition, but a more realistic explanation is simply that searchmysite.net is being drowned out by SEO spam.

Little has changed since then. In fact, in the searchmysite.net retrospective and future plans published the following year, I noted that there had been multiple weeks where not one single real person had visited a single blog entry for the whole week⁴.

Conclusion: Is there anybody out there?

I’m not one for conspiracy theories, but it’s hard not to think that there is some truth to The Dead Internet Theory, i.e. the belief that the Internet is now empty and devoid of real people, and that everything is just bots talking to bots to generate content and clicks in order to get a share of the all-important and ever-growing advertising revenue⁵.

I don’t think we’re quite at that stage yet though. I know there are still some real people doing great things on the internet, whether other real people know of their existence or not. However, I am genuinely concerned that my project to try to connect these people may have had unintended negative consequences, like in those post-apocalyptic stories where the robots have taken over the earth and well meaning attempts to link pockets of human resistance are doomed to draw the robots to them too.

This time I’m really not sure what the solution is.

UPDATE (16 May 2022): See also the discussion on Hacker News at https://news.ycombinator.com/item?id=31395231.