Tagged Questions
3
votes
1answer
477 views
Why is BingBot causing so many 404 errors by removing letters from url's?
Since the ForceRecrawl problem went away Bing has come back with some new trends.
I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also ...
5
votes
1answer
528 views
Am I covering the major search engine spiders in my anti-crawl protection white list?
I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my ...
2
votes
1answer
180 views
Do all search engines obey nofollow rel attribute on hyperlinks
I have been informed that search engine support is inconsistent for the rel attribute with the 'nofollow' value.
Is this the case.
From my research Google, Yahoo and Bing seem to obey this attribute ...
