The spider tag has no wiki summary.
10
votes
3answers
2k views
Does the Google spider render JavaScript?
Does the Google spider knows how to render JavaScript, or only HTML?
8
votes
4answers
601 views
Preventing robots from crawling specific part of a page
As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
20
votes
4answers
751 views
Is Yahoo Search the same as Bing Search now?
We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which ...
3
votes
1answer
268 views
Is there any reason to allow Yahoo! Slurp to crawl my site?
I thought a year or more ago Yahoo! would be using another search engine for results, and no longer using their own Slurp bot. However, a couple of the sites I manage Yahoo! Slurp continues to crawl ...
