I'm working on an eCommerce website where we list products from multiple vendors, and the products a user can buy varies throughout the day based on a pre-set schedule.

For example, something might be available only after 2PM on Wednesdays, or only between 11 AM and 3PM on weekdays.

We don't want the search engine rankings to be strongly affected by the time of day that the crawler visits the site.

We also cannot show the user products that are unavailable - they can only be shown things they can buy based on the time they visited.

So is detecting the crawler and giving different results the best way to go? Or are there other options I'm not aware of? Are there SEO penalties to showing different content to the crawler, and if so are there any ways to mitigate that?

  • Showing different content to search engines than users will get you into trouble no matter what the reason. This is considered cloaking even if that is not your intent. Please know that Google does use networks other than their own to check for cloaking. This happens quickly and looks like regular user traffic so that you will never know it has happened. I am not sure what advice I would give you except to simply let search engines access your site as any user would. If a page no longer exists you can issue a 404 or 410. If a page changes, then so be it.
    – closetnoc
    Sep 11 '15 at 0:21
  • unavailable_after meta tag may helps, havent worked with it but it came to my mind when reading the question. googleblog.blogspot.de/2007/07/… && searchenginewatch.com/sew/news/2334932/… - it seems it boils down to show 404 if the tag is not applied
    – user20874
    Apr 8 '16 at 5:29
  • and on YT Matt Cutts: "What should sites do with pages for products that are no longer available? " youtube.com/watch?v=9tz7Eexwp_A
    – user20874
    Apr 8 '16 at 5:32
  • I was probably being unnecessarily vague - it's not a matter of a product that's "no longer available" in the more permanent sense - 404's are the clear-cut choice there. The problem relates to restaurant menus. Bacon and eggs are available to the user in the morning, not the afternoon, but Google needs to know about it regardless of when they hit the site. Good comments and answers though - thanks a lot guys!
    – Crias
    Jul 11 '16 at 20:32

I have to agree with closetnoc that google doesn't like cloaking, and at the random rate google scans web pages, showing two different web pages at the same URL that the google crawler can scan isn't a good idea.

What I would suggest is to create the HTML so that it indicates to the users that you sell products at different timed intervals in text that google can scan.

Next use AJAX technology. This means include Javascript that requests a file from the server to show the random item for sale. All the better if the item is only a picture that loads, that way, google can't really penalize you since the static textual content is the same.

Just keep in mind if you do this, then users will be making two requests to the server for the page. One to load the HTML code that includes text content and one to load the actual picture and/or HTML that shows details of the items.

Whatever you do, just make sure the user has a full understanding of what is going on when they visit your site.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.