Tag Info

New answers tagged


I will answer your question exactly as asked in the hope that it will persuade you to reconsider if the root cause of your problem is not better resolved in some other way. Is it possible to completely block a page from being crawled under any circumstance? Not in a typically configured web server. The robots.txt file is just a polite request by your ...

Top 50 recent answers are included