Ways to deter unwanted visitors

Actions taken to remediate this issue

Two questions immediately came to mind:

1) How to stop the bad bots from hitting the website?

2) Why didn't the request result in a page not found condition and a redirect to the 404 'page not found' page?

Actions taken:

1) First I changed the robots.txt file to only allow robots from Google, Bing and a couple of other legitimate search engine robots. I did this even though I was aware that the robots.txt file is only advisory.

2. The hosting company blocked visitors from a range of IP addresses that were identified as causing the problem. This was probably the most important action taken.

3. With some help from guys on the Weavers Space forum, I implemented a redirect in the htaccess file to force the type of requests we were seeing to go to the 404 page.

The other option was to force a redirect to the Home page but I felt (and the hosting company agreed) that forcing to the 404 page was the better option here as the bots might get discouraged by the 404 response code. Here is the alternative code to force to the Home page that wasn't implemented:

4. I also implemented a set of rules in htaccess to block a bunch of bad robots as shown on this website: https://perishablepress.com/ultimate-htaccess-blacklist/ although the bad bot in question was almost certainly not on this list.

Luckily this sort of event only happens infrequently. The last time I had a problem like this was in 2017.

Amended 23/03/25 - the forward slash in the 2 code boxes above has been replaced by "FS" to prevent unparsable data being shown in the structured data test.