All my Siteground sites went offline for a few hours on Saturday due to limits on the volume of traffic being broken, what Sitegound refer to as the number of scripts being run. In my case the limit is 40k per day.
I rapidly put all the sites through Cloudflare but the number of hits is still excessive even now. One site in particular is being hit excessively nikamusicaltheatre and from the Cloudflare analytics page I can see that the problem is the Yandex bot which is a Russian search engine bot. Its hitting this site 26k times in 24 hours. None of my other sites are being crawled like this one.
Searching the web I came up with this solution in the htaccess file to bar Yandex:
## Bar Yandex bot ##
SetEnvIfNoCase User-Agent "^Yandex*" bad_bot
Deny from env=bad_bot
and also this one:
SetEnvIfNoCase User-agent “Baidu” spammer=yes
SetEnvIfNoCase User-agent “Yandex” spammer=yes
SetEnvIfNoCase User-agent “Sosospider” spammer=yes
<Limit GET PUT POST>
deny from env=spammer
22nd August Update
Above solutions seem not to work. Now up to 46k hits per 24hrs.
Now trying this code instead:
SetEnvIfNoCase User-agent (yandex|baidu|foobar) not-allowed=1
Allow from ALL
Deny from env=not-allowed
And also blocking by IP address all Russian IP addresses
and this in robots.txt file:
Disallow: / # blocks access to whole site
23rd August 2017
Well that worked. Siteground stats back to ground level.
Cloudflare stats show that Yandex is still hitting the site and that Russia has 65k hits so thats got me wondering about the relationship between what Cloudflare sees and the hosting but at least its sorted the immediate bandwidth problem at Siteground.