Tagged Questions

2
votes
3answers
303 views

User-agent identification and SEO crawler database

I am requested to analyze the traffic log of a site. In particular I have to identify the crawlers starting from the collected user agent values. I know there are 'trap' links that you can use to ...
7
votes
2answers
1k views

Ask.com Sitemap Crawler Down (for good)?

I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
3
votes
1answer
474 views

Why is BingBot causing so many 404 errors by removing letters from url's?

Since the ForceRecrawl problem went away Bing has come back with some new trends. I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also ...
2
votes
2answers
121 views

Does Yahoo Slurp crawl hammer a site?

A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
3
votes
4answers
316 views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
2
votes
3answers
738 views

What user agent should I set?

There is Ask bot, which sets this header: Mozilla/2.0 (compatible; Ask Jeeves/Teoma) Considering this, I have the following questions: If I'm writing a web crawler named Goofy, what user agent ...