All Questions
2 questions
0
votes
1
answer
668
views
Is it a good practice to block spam bots by IP ranges?
I'm getting too many hits on my website from Yandex. It doesn't obey robots.txt and by some way it can bypass .htaccess rules. So I'm thinking to block all Yandex IP ranges in my system firewall.
Is ...
- 257
5
votes
2
answers
4k
views
Grapeshot crawler ignoring robots.txt
Has anyone come across a crawler called Grapeshot? They are hammering the same page repeatedly on our website. I believe they are looking for ad related keywords, based on previous content ad ...
- 385