2

If you search for Cristiano Ronaldo on Google, one of the search results is his Instagram page. When you click on that search result, you are met with a login wall (at least on desktop). You cannot view any details from an Instagram page without logging in first, as reported here and here and here.

screenshot

I thought that Google did not show search results that link to pages behind a login wall. Is my understanding incorrect? How does Instagram convince Google to show its pages in search results, despite the login wall?

The reason that I'm asking is that I am considering developing a login wall for my own website that I am developing. It is important to understand how would treat a website with a login wall.

6
  • 1
    Instagram doesn't have a "login wall" i.stack.imgur.com/BvPDY.png Mar 9, 2021 at 9:39
  • @StephenOstermiller I've edited the question to make it clearer that Instagram does have a login wall. I'm not sure why you are not experiencing it, maybe Instagram is doing an A/B test.
    – Flimm
    Mar 9, 2021 at 10:28
  • 1
    More accurately, Instagram's login wall doesn't prevent you from viewing that page when clicking from the Google search results. Mar 9, 2021 at 10:49
  • @StephenOstermiller It does for me, on a guest profile on desktop Chrome with no extensions. Although, on mobile, there does not seem to be a login wall.
    – Flimm
    Mar 9, 2021 at 11:46
  • Here is a more full test using nine different popular browsers on different operating systems: app.crossbrowsertesting.com/public/i2a11f6a3a18c91f/screenshots/… The test just hits the URL on instagram without clicking from the Google search results. Only one of the browsers got the pay wall. Mar 9, 2021 at 12:00

1 Answer 1

1

Google does not have credentials to login to any service or account. That is the same reason they cannot index any message in Yahoo! Mail, simply because Googlebot does not have an account with Instagram or any other service. That is why Google cannot index anything behind access control, be it paid access or logged.

Googlebot is identifiable though. It has its user agent and Google publishes instructions to validate it . A site that to have its content indexed by Google but not generally accessible simply needs to serve content to Googlebot. The server needs to be configured to allow traffic when the Googlebot user agent is detected and (since that can be spoofed) can cross-check following Google-provided instructions which can be automated.

Websites that want Google indexing, simply need to let Googlebot through and they can control which pages are indexable of not this way dynamically.

2
  • Simply allowing Google access (and blocking users) isn't necessarily sufficient since this could be seen as cloaking. Google provides a method to indicate "paywalled" content using structured data to avoid the cloaking pitfall. Google extends its guidance with Flexible Sampling - where a certain amount of "free" content is made available to everyone (which would seem to be more of what Instagram is doing).
    – MrWhite
    Jun 23, 2021 at 15:57
  • This is what they advise on the very page you linked: If you want Google to crawl and index your content, including the paywalled sections, make sure Googlebot, and Googlebot-News if applicable, can access your page.
    – Itai
    Jun 23, 2021 at 17:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.