Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm trying to use httrack to mirror my blog, which is currently hosted on blogger. Problem: in spite of the robots.txt file, httrack tries to download everything in the /search subdirectory. This leads to an infinite regress of searches on searches.

Here's the robots.txt file (I've replaced my blog name with "myblog"):

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://myblog.blogspot.com/feeds/posts/default?orderby=updated

I can limit the crawl to depth 3 or 4, but I still get tons of search*.html and search/label/*.html files in the mirrored directory.

httrack claims to follow robots.txt. Why doesn't it work here? What can I do to fix it?

share|improve this question

migrated from superuser.com Jan 5 '12 at 16:28

This question came from our site for computer enthusiasts and power users.

1  
Just because you didn't mention it: Did you make sure you didn't disable the following of robot rules for the project and forget? It's in the Spiders tab of the options. –  Su' Jan 5 '12 at 16:40
    
Good question. I just ran httrack using the default settings, plus -r4 to limit the recursion depth. –  Abe Jan 5 '12 at 16:56

1 Answer 1

I don't know for sure, but maybe httrack is reading your 'Allow' rule as overriding the 'Disallow' rule.

You should remove the "Allow" rule regardless, as it is useless. User agents will crawl everything by default. You blocked the search directory, that is all that's required.

share|improve this answer
    
Good suggestion. In this case, I don't have control of the contents of robots.txt, so I can't modify the file. –  Abe Jan 6 '12 at 2:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.