Most of the sitemap generators I've found all scan my internal files and base it on that. However using apache and htaccess I have many aliases for things for example:

/brand/model/photos/ is empty directory wise, but on the web it relocates it to a zenphoto gallery using htaccess.

http://www.xml-sitemaps.com/ does What I need, but I'd like to have more control over it and I need it to be dynamic and run weekly or daily. Allow for specific change frequencies based on the file/directory. I also want to be able to make a "pretty" sitemap html file.

Counting the photo gallery I have around 300 links, without the photo gallery more like 100.

I have MySQL, PHP 5.3 installed.


I am using Debian Linux, so I'll need a crawler that can be run with cron or automated through linux.

  • You can edit you own posts any time you want an add or remove tags as you feel fit. In general you want to use already existing tags and only create new ones when it seems obvious that others will use it. Commented Feb 25, 2011 at 13:31
  • @RandomBen I felt "generator" was a tag that should be created based on the posts about "sitemap generators" and other generators.
    – ParoX
    Commented Feb 25, 2011 at 22:30

3 Answers 3


The best solution is to just generate dynamic sitemaps server-side. You have access to all of the application code and rewrite rules, so it doesn't really make sense to use a spidering tool to crawl your site. zenphoto has a plugin for generating sitemaps, as do most other CMSes. Just use a sitemap index to link to those sitemaps.

But if you still want to use a webcrawler, there are a few listed here.

Some people at Google have also developed a pretty ingenious sitemap tool called the Google Sitemap Generator. Instead of spidering the site (which is just the same thing that Google/Bing/Yahoo already do), it monitors server traffic to see how users access your pages and creates sitemaps that way.

  • I accepted this answer based on the long list to choose from, of which I am sure one will suit me.
    – ParoX
    Commented Feb 25, 2011 at 22:33

GSiteCrawler is my go to tool for this. You can schedule crawls and have it automatically upload the new files.

And it can generate a host of other useful reports.


You could consider trying A1 Sitemap Generator. Just make sure to uncheck "simplified easy mode". Then you can control the crawler with the output and analysis filters handling URLs matching with strings, paths, regular expressions, file extensions and mime types.

You can automate sitemap generator using scheduler / command line.

Not the answer you're looking for? Browse other questions tagged or ask your own question.