1

My website crashes my server when a certain page is crawled. I've tried implementing a robots.txt file to disallow that page and to implement a crawl delay, but web crawlers are ignoring it.

Is it possible to completely block a page from being crawled under any circumstance?

Improve this question
4
  • Are you using a platform/CMS/ecom? What kind of page is it? Why is it crashing your server? How are the bots finding it to begin with? Please include more information about this situation :)
    – dhaupin
    Commented Jun 9, 2015 at 18:25
  • I would be more concerned over why a single page crashes your server. That is seriously not good!! Start there. A bot is just a user. Even if no other bot ever touched that page, any user will crash you server. Solve that problem.
    – closetnoc
    Commented Jun 9, 2015 at 18:38
  • "but web crawlers are ignoring it." - have you been able to identify these "crawlers"? Do you know the IP addresses and/or user agents these bots are using? Presumably your server is OK when you have normal visitors? What's the difference?
    – MrWhite
    Commented Jun 9, 2015 at 18:39
  • "implement a crawl delay" - Where / how have you implemented this?
    – MrWhite
    Commented Jun 9, 2015 at 23:50

1 Answer 1

Reset to default
2

I will answer your question exactly as asked in the hope that it will persuade you to reconsider if the root cause of your problem is not better resolved in some other way.

Is it possible to completely block a page from being crawled under any circumstance?

Not in a typically configured web server.

The robots.txt file is just a polite request by your server. No one has to pay any attention to it. The big respectable commercial crawling bots do obey it voluntarily but there are many many others that do not.

There are 2 basic ways to control access to your server

1

Allow everyone access by default and block a subset based on some combination of IP address, how they identify themselves, frequency of usage etc etc. This is the typical configuration of the vast majority of public web sites. It requires you to know enough information about the visitor to uniquely identify traffic from them. This may not always be possible.

2

Block everyone by default and only allow a subset based on the above criterion and things like cookies or client certificates in the browser. Arguably this is not really a "public" web server anymore. It requires you to have a way of identifying ALL valid visitors automatically. Obviously this is impossible if you want any of the billions of normal web users to visit the site.

So you can completely block all possible bots using option 2 but you would then need some way to grant access to the visitors that you DO want to visit the site. This might be exactly what you intend (hard to guess without more context).

If option 2 is not what you want then you need to address the root cause of the crashing as the commenters have also encouraged you to do.

A separate question explaining the technical details of the crash and how to prevent it would be a helpful way to proceed.

Improve this answer
1
  • 1
    My family members like to attack symptoms rather than solving the problem so I am super-used to these kinds of questions. I can't help but scream at the screen "Solve the &^%^%$%$#%@#ing problem!" ;-) My system (web) can be a bit sensitive if you goof up just a little bit- this is because of the heavy automation. I am used to digging deep into researching problems which can be rather obscure sometimes and several layers deep. This is something the OP will just have to roll-up his sleeves and root-around for the cause of the problem and solve.
    – closetnoc
    Commented Jun 10, 2015 at 1:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.