I'm trying to use httrack to mirror my blog, which is currently hosted on blogger. Problem: in spite of the robots.txt file, httrack tries to download everything in the /search subdirectory. This leads to an infinite regress of searches on searches.
Here's the robots.txt file (I've replaced my blog name with "myblog"):
User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: http://myblog.blogspot.com/feeds/posts/default?orderby=updated
I can limit the crawl to depth 3 or 4, but I still get tons of search*.html and search/label/*.html files in the mirrored directory.
httrack claims to follow robots.txt. Why doesn't it work here? What can I do to fix it?