Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal code or city through drupal and search query is outputted through aspx. The problem is that google/bing keeps indexing the aspx returned query pages and then we receive 404 errors.

How can I prevent googlebot from indexing the web service. We are on IIS 7.5. Do we setup a rule on the server? I have tried blocking it in robots.txt but am unsure if I have the correct syntax.

Disallow: /*?
User-agent: Googlebot
Noindex: /office/default.aspx
Noindex: /?keyword
Noindex: /el​ectronics/locationdetails.aspx?
Noindex: /electronics/communitydetails.aspx?
Noindex: /electronics/Communitylist.aspx?
Noindex: /office/communitysearch.aspx?
Noindex: /Electronics/Communitylist.aspx?
Noindex: /Office/LocationDetails.aspx?
share|improve this question
add comment

2 Answers

Your robots.txt file does not appear to conform to the robots.txt specification (you must specify the user agent before the directives and I do not believe that Noindex is a valid directive).

Consider the following updates:

User-agent: googlebot
Disallow: /Office/LocationDetails.aspx
Disallow: /office/default.aspx
Disallow: /Electronics/Communitylist.aspx
Disallow: /electronics/locationdetails.aspx
Disallow: /electronics/Communitydetails.aspx
Disallow: /electronics/communitydetails.aspx

... alternatively, you could configure your webserver (or application) to add noindex/none X-Robots-Tag headers where appropriate as described in detail at Google's Robots meta tag and X-Robots-Tag HTTP header specifications document.

share|improve this answer
add comment

Use "NoINDEX, NOFOLLOW" Meta Robots tag in each pages under the head tag or certain directives under robots.txt will do the job for you.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.