Tagged Questions
5
votes
1answer
528 views
Am I covering the major search engine spiders in my anti-crawl protection white list?
I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my ...
1
vote
1answer
428 views
How does bing-bot( is that the right spider-name? ) and googlebot interpret 301 redirect?
I've been looking for documentation on how the Microsoft and Google bots interpret 301 redirects.
It seems that google-bot stores documents on a url based index system. But I haven't been able to ...
2
votes
1answer
218 views
Should I strategically use NoFollow on internal links to make our indexing crawl more efficient?
We have a well designed internal linking system that makes certain no page is more than four pages deep, and the indexing process has gone well for us.
However, in addition to the hierarchical page ...
10
votes
3answers
2k views
Does the Google spider render JavaScript?
Does the Google spider knows how to render JavaScript, or only HTML?
