New answers tagged


From your comments on another answer, MJ12Bot is visiting your site less than once an hour (421 times in 25 days.) The best thing to do is to not worry about it. Crawl-Delay is useless for you because no crawler will obey a craw-delay so large.


It appears you're using query strings on a PHP page to generate the file links. Try adding a wildcard to the end of your ALLOW, as this will form part of the URI. Also, have you checked the URL parameters section to see how Google is treating these variables? You can also explicitly set behaviour, make sure Google understands how to use the query strings ...


There are really 2 issues here: Will the robots.txt on your site Disallow (block) Wayback from crawling your site. Will Wayback crawl your site. For point #1: As others have said, the correct entry for robots.txt is: User-agent: ia_archiver Disallow: Keep in mind that it might take a while (perhaps a good long while), for Wayback to notice any changes ...

Top 50 recent answers are included