Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've got a sitemap that is generated daily with a lot of links to product pages. These products are imported daily from another data source. Because the update consists of throwing away all current product info and replacing it with the new imported info the last modified date always jumps one day. This is also used in the sitemap. Even for products that haven't changed. All product pages pretend to have been updated.

Will Google penalize the website for pretending the pages have changed from day to day while they haven't?

My solution would be to only change the entry only if the new imported product data differs from the previous data. I just want to make sure this is a useful upgrade to make, while I could also spend my time on other improvements.

share|improve this question
up vote 4 down vote accepted

I've never heard anything about a penalty due to this. At worst you're wasting the spider's time, but that's part of why we have computers in the first place: doing tedious repetitive things. Still, you should ideally be addressing the issue.


My solution would be to only change the entry only if the new imported product data differs from the previous data.

...is what you should be doing in the first place, regardless of external considerations like sitemaps, etc. If your content isn't different(and I would include deleting and replacing with identical information in that description), then your lastmod date shouldn't be modified. Here you're wasting your own resources. You haven't said how many products are involved, but there's going to be a point where this process is going to get slow and computationally expensive.

share|improve this answer
I totally agree. However, I'm dependable on another company that delivers the data. They always send every product (+200) in their data exports. So updating the lot seemed the best solution a few years ago. My client doesn't have the budget to solve this properly. These ex / imports happen at night, so the extra used resources are not a big problem at the moment. – Elicit Feb 13 '12 at 10:14

No. Google will use lastmod as a hint (same as all sitemap values) but if it decides that your content is not getting updated daily then it will simply ignore it and revisit your pages on its own schedule.

share|improve this answer

Google will not penalize you for this. In order to get a penalty you really need to go black hat on Google's ass so don't worry about that. Google will find out soon enough if you're content has changes (that's what they've been working on the past few years) and use the lastmod property as a hint.

share|improve this answer

I don't work for Google, and can't say for sure what they actually do, but the sensible way for them to treat <lastmod> timestamps would be as hints not to waste time re-crawling pages that haven't changed.

So if you report all your pages as changed every day, Googlebot will just keep crawling all your pages in whatever order it feels like, rather than only focusing on the pages that have changed. In effect, it's just as if you didn't report any last modification timestamps at all.

The main reason to provide correct <lastmod> timestamps is to make changes to your site show up faster in Google's index. If you have hundreds of pages on your site, it's going to take a while for Google to crawl them all and find any changes. However, if you tell Googlebot which pages have changed recently, it can crawl those pages first and avoid wasting so much time on the rest.

Of course, you could just bump up Googlebot's crawl rate in Webmaster Tools instead and hope for the best. But really, it shouldn't be too hard to make your update script preserve timestamps. For example, I assume you're currently doing something like this:

for each product do:
    write new page content into product page file;
end do;

If so, just change it to something like this instead:

for each product do:
    read old page content from product page file into string A;
    write new page content into string B;
    if A is not equal to B then:
        write string B into product page file;
    end if;
end do;
share|improve this answer

No it simply ignores the information you have provided when it is incorrect. In this case, web crawlers figure out by themselves how often they should crawl your pages.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.