I had a very simple robots.txt file setup for a site I maintain. After a spike of traffic that the ISP put down to crawlers they suggested I add a crawl delay directive which is fair enough. So I ended up with this file
I still receive spikes in traffic that are causing downtime. The ISP told me in relation to the robots file I had setup (shown above) that "the crawl delay is configured only for the '/a-page-i-wanted-to-ignore ' page and not for the entire website."
I wanted to check, is that correct? If a command like crawl-delay placed under a 'disallow' line will it be specific to the disallow clause ?