I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal code or city through drupal and search query is outputted through aspx. The problem is that google/bing keeps indexing the aspx returned query pages and then we receive 404 errors.

How can I prevent googlebot from indexing the web service. We are on IIS 7.5. Do we setup a rule on the server? I have tried blocking it in robots.txt but am unsure if I have the correct syntax.

Disallow: /*?
User-agent: Googlebot
Noindex: /office/default.aspx
Noindex: /?keyword
Noindex: /el​ectronics/locationdetails.aspx?
Noindex: /electronics/communitydetails.aspx?
Noindex: /electronics/Communitylist.aspx?
Noindex: /office/communitysearch.aspx?
Noindex: /Electronics/Communitylist.aspx?
Noindex: /Office/LocationDetails.aspx?

2 Answers 2


Your robots.txt file does not appear to conform to the robots.txt specification (you must specify the user agent before the directives and I do not believe that Noindex is a valid directive).

Consider the following updates:

User-agent: googlebot
Disallow: /Office/LocationDetails.aspx
Disallow: /office/default.aspx
Disallow: /Electronics/Communitylist.aspx
Disallow: /electronics/locationdetails.aspx
Disallow: /electronics/Communitydetails.aspx
Disallow: /electronics/communitydetails.aspx

... alternatively, you could configure your webserver (or application) to add noindex/none X-Robots-Tag headers where appropriate as described in detail at Google's Robots meta tag and X-Robots-Tag HTTP header specifications document.


Use "NoINDEX, NOFOLLOW" Meta Robots tag in each pages under the head tag or certain directives under robots.txt will do the job for you.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.