0

We gave a client the recommendation to add an item to their robot.txt (see below) to allow Google to crawl the full site. The idea was that this would allow them to crawl and index any pages that aren't excluded by a noindex tag.

User-agent: Googlebot Disallow:

User-agent: Googlebot-image Disallow:

however, the developer feels they should remove the command from the robot.txt file to prevent Google from re-indexing customer pages and avoid receiving false positive notifications. This change is admin config only and can be monitored and they want to know if they should go ahead and do this.

The issue is on this URL - https://www.example/review/product/listAjax/id/124/?limit=50 that keeps on getting crawled and indexed by Google. We had recommended they remove this from robot.txt so that the Google bots wouldn't crawl everything and that caused huge SEO problems where Google decided to index the login & query pages it's blocked from crawling, now that it can't read the 'noindex' tags anymore (the internet says Google will rarely index blocked pages, unless they are really popular/high demand links.

so, removing these lines will prevent Google from indexing the ajax URL https://www.example.com/review/product/listAjax/id/*** <https://www.example.com/review/product/listAjax/id/***>

So should we remove these? What is best practice? Can regular expression matching regex be used prevent crawling of ajax pages in robot.txt? What should I recommend?

1 Answer 1

0

Robots.txt is NOT updated on every request, thus, it's possible for Google or other search engines to have a stale version of it, crawling and indexing unwanted pages on your site. You need to place the restriction on each page you don't want crawled in addition to the Robots.txt file.

As Google says:

To properly prevent your URL from appearing in Google search results, password-protect the files on your server, use the noindex meta tag or response header, or remove the page entirely. https://developers.google.com/search/docs/crawling-indexing/robots/intro

Therefore, all your administrative pages should send the noindex X-Robots-Tag, either in the HTML or as a Server Header, where you can configure it using regular expressions in htaccess or IIS.

In addition, your robots.txt should end like this:

User-agent: *
Disallow: /review/product/listAjax/id/
Disallow: /put-your-admin-directory-here/

This will prevent indexing any files under those directories, but still allowing the rest of the site to be indexed.

1
  • Thank you so much Luis that's very helpful info. Jan 10 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.