up vote 2 down vote favorite
1
share [g+] share [fb]

I'm trying to use httrack to mirror my blog, which is currently hosted on blogger. Problem: in spite of the robots.txt file, httrack tries to download everything in the /search subdirectory. This leads to an infinite regress of searches on searches.

Here's the robots.txt file (I've replaced my blog name with "myblog"):

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://myblog.blogspot.com/feeds/posts/default?orderby=updated

I can limit the crawl to depth 3 or 4, but I still get tons of search*.html and search/label/*.html files in the mirrored directory.

httrack claims to follow robots.txt. Why doesn't it work here? What can I do to fix it?

link|improve this question
1  
Just because you didn't mention it: Did you make sure you didn't disable the following of robot rules for the project and forget? It's in the Spiders tab of the options. – Su' Jan 5 at 16:40
Good question. I just ran httrack using the default settings, plus -r4 to limit the recursion depth. – Abe Jan 5 at 16:56
feedback

migrated from superuser.com Jan 5 at 16:28

This question came from our site for computer enthusiasts and power users.

1 Answer

up vote 2 down vote

I don't know for sure, but maybe httrack is reading your 'Allow' rule as overriding the 'Disallow' rule.

You should remove the "Allow" rule regardless, as it is useless. User agents will crawl everything by default. You blocked the search directory, that is all that's required.

link|improve this answer
Good suggestion. In this case, I don't have control of the contents of robots.txt, so I can't modify the file. – Abe Jan 6 at 2:08
feedback

Your Answer

 
or
required, but never shown

Not the answer you're looking for? Browse other questions tagged or ask your own question.