Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I currently am tasked with submitting our website sitemap to the search engines every week.

We have a module which does offer sitemap generation but we find using it does not work very well as not all pages are included and it does not split the sitemap by content.

I've used various (online and offline) tools to generate the sitemaps which is not the problem. The problem is that after every generation (which takes most of each Monday) I have to manually go through the sitemap and categorise the links in to products, pages, categories and sub categories.

I've experimented successfully with XSL to split the sitemap but it is still a labour intensive process.

Does anyone know of a good method to split the sitemap?

Currently there are around 20,000 links (iirc) in total.

share|improve this question
    
Why do they need to be in different sitemaps? Each sitemap file can be 50MB and hold 50,000 URLs. The URLs will get submitted to search engines just fine in a single sitemap. –  Stephen Ostermiller Jan 6 at 16:35
    
Are you trying to split the sitemap based on a URL pattern? If so, please provide some examples. –  Stephen Ostermiller Jan 6 at 16:36
    
When your sitemap is generated, is each URL on a single line by itself? If so, it would be pretty trivial to split the file with a small Perl script. If entries span multiple lines or there are multiple entries per line, then scripting languages may be of slightly less value. –  Stephen Ostermiller Jan 6 at 16:38
    
It is something which our SEO guys have forwarded to me to do. Apparently it is advantageous for indexing speed and will be helpful if something goes wrong in pinpointing the problem. A pattern could be used for some, for example weburl.co.uk/category/subcategory/product but this would not work as across the board. Also the sitemaps I have generated up to now have all been formatted so each URL is on multiple lines. –  James Jan 6 at 17:01
    
If a pattern won't work for all the URLs, what criteria do you use to split the file when you do it by hand? –  Stephen Ostermiller Jan 6 at 18:16

3 Answers 3

up vote 0 down vote accepted

There are not off the shelf products that are going to do what you want. You could write a script to do it. It wouldn't be too hard in a language like Perl or Python.

If you sitemap entries were on single lines, it would be much easier. You will need to hook up an XML parser instead of just reading the file line by line.

Once you can read the file line by line in the script, you place them in various buckets using the same criteria that you use to do so by hand.

share|improve this answer
    
Thanks Stephen. I have no experience with either of those languages but I'm sure I can figure something out, I'll definitely look in to it. I would +1 your answer but my rep is too low, I'll give others a chance to answer before marking it as 'the' answer also. :) –  James Jan 10 at 16:39

I can't think of anything a sitemap is used for that requires content split up by content type. Can you dig into that a bit and give us a more detailed picture of what is being created, how it is being used, and why it would want to be edited?

The big worry is the pages that aren't being generated--that is content that Google and other search engines aren't seeing (or are seeing only to the extent that they follow links from pages in the site map). That would be where I would spend time before anything else.

Related to this is the possibility that you are also using a variant of your site map to help people find things on your site. That is usually not useful with a site as large as yours, which would raise the question of how people navigate, what are they looking for, how do you convey a "scent of information," etc.

We often think of websites as places where people land on the home page and then wander about. In reality, they land on the page that a search engine or link suggested would answer a question or meet a need, and then do some minor exploring (or, most often, go on to another website). So, I also raise the possibility that there is an architecture and/or SEO issue--but that takes us far afield from the question as it was asked, and gets into analytics telling you how people are using the site, and then using that as a guide for further development.

In short--a site map is a shortcut invented to help search engines find your content. Need to understand better why you aren't getting a whole site listed, and what you are hoping to accomplish by editing.

share|improve this answer
    
Since I asked this question the requirement has gone. They seem to have abandoned the idea, which makes me wonder if they came to the same conclusion as you. (I am but a lowly minion and don't get told these things (: ). I do know though that most of our traffic is directly on to product pages and not many actually enter our category pages (something which we are working on a remedy for (although not such a bad thing?). –  James Aug 11 at 7:20

I would try to get it on this way.

Firstly ensure, that URLs are "speaking", so e.g. they look like domain.com/category1/item1, domain.com/category2/item2.

Then export all existing URLs into different files, which are sitemaps, by sorting URLs alphabetically, so each sitemap file gets URLs from single category (category1.xml, category2.xml)

It all can be done by simplest PHP-script, which reads all internal URLs and writes them into once created files. You can even create these sitemap-files manually, for each category.

Look here: http://php.net/manual/en/domdocument.save.php.

share|improve this answer
    
The URLs are indeed 'speaking'. I would have definitely tried your suggestion of a PHP script had I thought of it back then and been as comfortable with scripts as I am now. Thanks. –  James Aug 11 at 7:23
    
before you will be "comfortable with scripts" you can achieve it manually –  Evgeniy Aug 11 at 12:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.