Tag Info

New answers tagged


If the URLs they are requesting follow some patterns, I think it is a good idea to block those pages. Based on your comment above, in your robots.txt you could add: User-agent: * Disallow: /data/ That will work as long as you don't have any real URLs beginning with data. The second thing you may want to do is decrease Google's crawl rate. You can do this ...

Top 50 recent answers are included