All Questions
3 questions
1
vote
0
answers
55
views
Do any search engines exist which disobey rel="nofollow" or "Disallow"? [closed]
Moving beyond the big players - Google, Yahoo, Bing, Ask, DDG (who all obey "nofollow"s), are there any search engines out there which deliberately disobey rel='nofollow' links and Disallow rules in ...
- 121
1
vote
1
answer
155
views
How to prevent search engine bots from crawling specific pages?
My website crashes my server when a certain page is crawled. I've tried implementing a robots.txt file to disallow that page and to implement a crawl delay, but web crawlers are ignoring it.
Is it ...
- 235
2
votes
1
answer
2k
views
How to Keep "Good" Bots That Ignore Robots & Nofollow From Getting Into Honeypots?
We have a self generating honeypot that is designed to fill crawlers with junk data (among other things). Its protected with the right "treaties" and headers so 99% of good bots stay away. Today ...
- 3,339