3

For a couple of pages on our site, I'm writing a widget that relies on what it displays on other pages to determine what it displays on the current page. Basically, the purpose is just to ensure there's no duplicate content. It's not individualized per user, just depended on what's being displayed elsewhere at the current given time.

My CTO will not allow me to save the data in the database or even a log file to keep a persisting state, so to accomplish this the only other way I can think of is set a $_SESSION variable and store the persisting state there. However, I'm realizing google's bot probably doesn't use cookies so I'm not sure if this will work.

Does anyone know if Google will still index the pages if what they're displaying relies on a session variable? If not, is there another way to store a persisting state across pages that doesn't use the db or log file that googlebot will understand?

Improve this question
2
  • 1
    As to "is there another way to store a persisting state without a db" - take a leaf out of ASP.Net WebForms book and take a look at "ViewState" - basically shoves all the state into an encrypted string in a form, and treats all page navigation as POST. Messy, but works.
    – Moo
    Apr 5, 2019 at 0:51
  • @Moo does googlebot respect this? And how would I do that in PHP, all I can find is ASP.NET implementations.
    – Mitch
    Apr 5, 2019 at 2:03

3 Answers 3

Reset to default
3

Back in September 2018 John Mueller from Google tweeted:

@jakebohall tweets: "@JohnMu Any chance you'd share a few examples/ideas of when/why Googlebot uses cookies? I haven't come across anything recent on this other than WRS clearing them across page loads..." @JohnMu replies: "We almost never use them. If you need cookie support to view a page, it's almost certain that we wouldn't be able to index it. If you notice it in an audit, fix it (remove the dependency), don't assume that maybe it'll work :-)."

Also see:

Source: https://www.seroundtable.com/google-cookies-seo-26344.html

Google's John Mueller said on Twitter that Google almost certainly cannot index a page that requires cookies. He said if you want Google to index the page, make sure to "remove the dependency" on cookies.

Improve this answer
1
  • So is there another way to store a persisting state so when googlebot crawls one page I can ensure there won't be any duplicate content on the next page they crawl?
    – Mitch
    Apr 5, 2019 at 18:28
1

What do you do when a user visits the site for the first time? Presumably you calculate what needs to be displayed, display it, and "cache" something in the session (as you mention). Every Googlebot visit is like the user's first time visit (as @Simon mentions - the Googlebot does not use cookies, so no session data can persist).

So, assuming you do display this content to the user on their first visit then GoogleBot will also see this content, except that it will need to be calculated (which could be slow?) on every request.

Improve this answer
3
  • Yes, but does this mean it treats every page as a first time user? The calculation cost is what it is, but if a user sees one page then sees the other page it needs to not have duplicate content so I need to know what was on the first page to ensure that. So according to John Mu a session variable wouldn't work, but would the google bot just not be able to index the page at all? I want to leave the code in so at least I can say I wrote the checks whether googlebot actually cares or not is fine by me as long as they can still index the page. Will storing a variable in session block indexing.?
    – Mitch
    Apr 5, 2019 at 2:01
  • Yes, every page the Googlebot visits is seen as a first time user (assuming your session storage is based on a cookie). You could potentially track the (Google)bot based on some hash (IP + User-Agent perhaps), but you say you're unable to store additional information in a server-side database/log. But this may not be particularly accurate anyway. Googlebot doesn't necessarily "crawl" in a consistent manner - like a user - it builds a list of URLs to visit and these could be visited in any order.
    – MrWhite
    Apr 8, 2019 at 0:46
  • If Googlebot crawls (and indexes) different content to what a user sees when visiting the same URL then that is also a problem (and could be considered cloaking). Indexing is based on the URL... one URL = one page of content.
    – MrWhite
    Apr 8, 2019 at 0:48
1

Use a hash of the UTC date and the ip address, then use the hash as a seed to a random number generator, use the random number generator to generate a permutation of which content goes to which page.

Results: random page content that varies over time while being static per user, unique content per page, no state stored anywhere (not in cookies, not in session url parameters, not in db, etc.), compatible with all search engines.

downside: you need a static list of all the urls in order to ensure that each url has a unique piece of content. Every page request would have to map content to each url in the same random pattern (based on the static seed hashed from the date+etc.). Doing this requires processing time that is linearly proportional with the number of urls.

Improve this answer
4
  • That's exactly how it works now with just the date, but even if I hash their IP aswell that still wouldn't help with the problem, which is that after a user goes to another page what it displays there is dependent on what they saw on the original page to ensure no duplicate content
    – Mitch
    Apr 5, 2019 at 18:26
  • I should add that there are content pools the randoms are pulling from, these could have overlapping content and aren't in a guaranteed distribution so even with different arrays of randoms duplicate content is unlikely but not impossible & I need guaranteed unique content as a specification for the task.
    – Mitch
    Apr 5, 2019 at 18:45
  • @MitchellLewis I answered to quickly, the added downside is that you would have to have a static list of all the possible urls. This is required to ensure a unique mapping for each url.
    – Brenda.ZMPOV
    Apr 8, 2019 at 18:17
  • Yeah, & that's a problem since one of the key specifications was portability and that we don't know all the URLs we'll put this widget on, the only way to keep it portable and maintain this method would be to store any new URL that it gets called from in the DB but my CTO won't let me store anything
    – Mitch
    Apr 9, 2019 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.