I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal code or city through drupal and search query is outputted through aspx. The problem is that google/bing keeps indexing the aspx returned query pages and then we receive 404 errors.
How can I prevent googlebot from indexing the web service. We are on IIS 7.5. Do we setup a rule on the server? I have tried blocking it in robots.txt but am unsure if I have the correct syntax.
Disallow: /*? User-agent: Googlebot Noindex: /office/default.aspx Noindex: /?keyword Noindex: /electronics/locationdetails.aspx? Noindex: /electronics/communitydetails.aspx? Noindex: /electronics/Communitylist.aspx? Noindex: /office/communitysearch.aspx? Noindex: /Electronics/Communitylist.aspx? Noindex: /Office/LocationDetails.aspx?