Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Most of the sitemap generators I've found all scan my internal files and base it on that. However using apache and htaccess I have many aliases for things for example:

/brand/model/photos/ is empty directory wise, but on the web it relocates it to a zenphoto gallery using htaccess.

http://www.xml-sitemaps.com/ does What I need, but I'd like to have more control over it and I need it to be dynamic and run weekly or daily. Allow for specific change frequencies based on the file/directory. I also want to be able to make a "pretty" sitemap html file.

Counting the photo gallery I have around 300 links, without the photo gallery more like 100.

I have MySQL, PHP 5.3 installed.

Update:

I am using Debian Linux, so I'll need a crawler that can be run with cron or automated through linux.

share|improve this question
    
You can edit you own posts any time you want an add or remove tags as you feel fit. In general you want to use already existing tags and only create new ones when it seems obvious that others will use it. –  RandomBen Feb 25 '11 at 13:31
    
@RandomBen I felt "generator" was a tag that should be created based on the posts about "sitemap generators" and other generators. –  ParoX Feb 25 '11 at 22:30

3 Answers 3

up vote 3 down vote accepted

The best solution is to just generate dynamic sitemaps server-side. You have access to all of the application code and rewrite rules, so it doesn't really make sense to use a spidering tool to crawl your site. zenphoto has a plugin for generating sitemaps, as do most other CMSes. Just use a sitemap index to link to those sitemaps.

But if you still want to use a webcrawler, there are a few listed here.

Some people at Google have also developed a pretty ingenious sitemap tool called the Google Sitemap Generator. Instead of spidering the site (which is just the same thing that Google/Bing/Yahoo already do), it monitors server traffic to see how users access your pages and creates sitemaps that way.

share|improve this answer
    
I accepted this answer based on the long list to choose from, of which I am sure one will suit me. –  ParoX Feb 25 '11 at 22:33

GSiteCrawler is my go to tool for this. You can schedule crawls and have it automatically upload the new files.

And it can generate a host of other useful reports.

share|improve this answer

You could consider trying A1 Sitemap Generator. Just make sure to uncheck "simplified easy mode". Then you can control the crawler with the output and analysis filters handling URLs matching with strings, paths, regular expressions, file extensions and mime types.

You can automate sitemap generator using scheduler / command line.

share|improve this answer

protected by John Conde Feb 7 '13 at 12:39

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.