In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing.
So what I thought of; is depending on the server health (CPU Utilization), by which I can respond with 503 to the bots if the server is on heavy load, so they stop crawling and come back again after some time which I will specify in the "Retry-After" header.
My questions are:
- Will responding with 503 for any Robots.txt request stop the Googlebot (to be specific) from accessing all of my website resources?
- Do I need to respond with 503 for all of my website resources, and if I did that, will it stop Googlebot from retrying, and will it obey the "Retry-After" header