We gave a client the recommendation to add an item to their robot.txt (see below) to allow Google to crawl the full site. The idea was that this would allow them to crawl and index any pages that aren't excluded by a noindex tag.
User-agent: Googlebot Disallow:
User-agent: Googlebot-image Disallow:
however, the developer feels they should remove the command from the robot.txt file to prevent Google from re-indexing customer pages and avoid receiving false positive notifications. This change is admin config only and can be monitored and they want to know if they should go ahead and do this.
The issue is on this URL -
https://www.example/review/product/listAjax/id/124/?limit=50 that keeps on getting crawled and indexed by Google. We had recommended they remove this from robot.txt so that the Google bots wouldn't crawl everything and that caused huge SEO problems where Google decided to index the login & query pages it's blocked from crawling, now that it can't read the 'noindex' tags anymore (the internet says Google will rarely index blocked pages, unless they are really popular/high demand links.
so, removing these lines will prevent Google from indexing the ajax URL
So should we remove these? What is best practice? Can regular expression matching regex be used prevent crawling of ajax pages in robot.txt? What should I recommend?