We have a serious traffic issue on our site and we want to eliminate crawlers as part of the problem. If we disallow crawling using robots.txt only during some hours, will it hurt our SEO?
-
4Googlebot may only fetch robots.txt once a day. It isn't possible to have it obey robots.txt on an hourly basis. Google's John Mueller warns about your approach here: webmasters.stackexchange.com/a/86429– Stephen Ostermiller ♦Jan 18, 2018 at 20:05
Add a comment
|
1 Answer
Focus your energy on improving your site/server instead of trying some hacky workaround. You never know what time of day search engines will crawl your site, so only allowing them during certain windows may have a hefty negative affect. If you block crawlers when Google comes along, you won't like the results.
What you can do for Google is set the crawl rate. Limiting the crawl rate ("how many requests per second Googlebot makes to your site when it is crawling it: for example, 5 requests per second") will help alleviate the load impact without completely blocking them. If you have a massive site then this may mean a longer delay before new updates are indexed, but it's better than dropping off entirely.
At least for Bing, you do have a bit more control. As per their how to page:
You can tell Bingbot to crawl your site faster or slower than the normal crawl rate for each of the 24 hours in the day. This way you can to limit Bingbot activity when your visitors are on your site and allow us more bandwidth during quieter hours.
answered Jan 18, 2018 at 20:02
