3

When we create an XML sitemap for a site with dynamic data, should we prepend all new data to the start of the sitemap OR append it to the end? Do search engines hate it when a sitemap is 100 % different each time OR when they see that it's mainly the same, but just a bit different at the end?

Example

Let's imagine we're creating an XML sitemap for Stack Overflow → a site with dynamic data.

Let's also assume that, ever day we have 1000 new 'items' to get added to the sitemap and we start out with 200,000 items.

I'm also generating the sitemap every hour or every day, don't really care just yet.

Now, with all these items to be sitemap'd, first I'll need to have a sitemap index that points to each page of items to get indexed. Kewl.

If I add the new items to the START of the list of items to get indexed, then each 'page' in the index will be different, compared to the previous time the search engine grabbed the sitemap index.

For example:

  • Day 1: Page1 will have 100 items, with the most recent item having a timestamp of 1pm, day1.
  • Day 2: page1 will now have a complete new set of 100 items, with the timestamp of the newest item in this page being 1pm day2.

This also means every other page, in the index is different to the previous.

So does this mean the search engine would need to go through EACH PAGE again to see what resources it needs to index? But at least all the new content is first, so the assumption here is that the search engine will be indexing the newest stuff, first.

As opposed to …

appending any new data to the end of a sitemap …

  • Day 1: Page1 has 100 items. most recent is timestampe 1pm, day1
  • Day 2: Page1 is still identical to page1, the day before. Now, page 22 has the most recent stuff, which didn't exist the day before.

Here I'm assuming the search engine will see all of this stuff and say 'ahh! most of these pages haven't changed since I was last here … so I'll ignore them … but let's see AH! there's a few new pages that DO have some new data … so let's check them out instead'.

Anyone have any experience with this?

Improve this question

1 Answer 1

Reset to default
4

It doesn't matter what order you put it in. The sitemaps just tell the search engines where your content is. They don't stop reading the file just because you've submitted the URLs in a previous sitemap or some content is newer/older then other content. They'll read the whole thing unless it is larger then the allowed size or there is an error in your XML.

Improve this answer
2
  • So you're saying that, if my sitemap has 200,001 items .. and the index lists 21 pages .. then the search engine will grab all those references and say .. ok .. out of all these 200,001 items, these ones i know have changed .. and these ones are all the same since the last time (because the sitemap has the timestamp, per node) .. and then decide if it wants to goto all the new/updated nodes?
    – Pure.Krome
    Commented Sep 1, 2011 at 1:20
  • Basically, yes. Since XML sitemaps are more suggestions then directives the search engines can do whatever they want with that information. But you're summary is pretty much how it works.
    – John Conde ♦
    Commented Sep 1, 2011 at 1:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.