Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

A certain page on our website keeps generating entries like the following on our IIS6 logs:

2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707368055555 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707277777777 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707347222222 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707291666666 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707263888888 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707326388888 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707437500000 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707451388888 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707340277777 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707270833333 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:35 W3SVC943830190 10.20.101.146 GET /abc workaround=407707381944444 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:35 W3SVC943830190 10.20.101.146 GET /abc workaround=407707284722222 80 - 66.249.85.65 - 200 0 0
2011-08-16 00:53:35 W3SVC943830190 10.20.101.146 GET /abc workaround=407707201388888 80 - 66.249.85.65 - 200 0 0

Doing a reverse DNS lookup, these IP addresses making these calls seems to be a Googlebot. What does "workaround=" on this line mean?

We always seem to be getting "workaround=" calls on 1 particular page only, sometimes multiple times within a few seconds like this example. What is the cause of this and how can I prevent it?

This specific page does make AJAX calls via jQuery to a 3rd party service and they are reporting a spike in calls being made and I am suspecting this bot calls to be the issue.

share|improve this question
    
See stackoverflow.com/questions/2434445/… - I'd go with a robots.txt solution myself to prevent bots from crawling a specific page if it's giving problems. –  stealthyninja Aug 16 '11 at 14:06
add comment

1 Answer

I'm not sure exactly what "workaround" means. It seems likely that the third party AJAX is including that string in data that is returned. Google may be finding it in either HTML or JavaScript content and then deciding that it looks enough like a URL that it should crawl it.

To prevent Googlebot from crawling those URLs on your site, you could add this to your robots.txt file:

User-Agent: Googlebot
Disallow: /*workaround=

It sounds like the AJAX shouldn't be crawled either. In that case, the third party site might have to add something to their robots.txt. You could also put the JavaScript code that calls the AJAX into robots.txt so that Googlebot can't see it:

User-Agent: Googlebot
Disallow: /js/call-third-party-ajax.js
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.