1

I have a site that is login-blocked/password-protected where Googlebot crawling has worked for 1+years and suddenly stopped. enter image description here

I am trying to figure out how this could have happened. It is a large site with a large code base. I know about 1 year ago I verified that URL-live-test was properly rendering pages with all details as I was investigating a rendering issue in the public sister site and validated that this one was working properly.

As far as I know there was never any code to serve one version of the site to crawlers and another to users.

2
  • What's the website written in? PHP?
    – Laurel
    Feb 24 at 20:28
  • Angular single page app
    – James L.
    Feb 24 at 21:07

1 Answer 1

2

If the pages were password protected, yet accessible to the crawlers, then you were probably serving different pages to the two. The timing tells you something. It happened when you confirmed that it was working and when started your sister site.

The primary suspect is that something has changed in your code (that did not get tested). You did a live test a year ago. Do it again. Prove to yourself that the crawlers are seeing a fully rendered page.

The second suspect is that the pages are regarded as a duplicate of the sister site. Explore the reports in Google Search Console to confirm the reason.

3
  • Yup. Either this or there was an exception in the login blocking/password protection (eg looking at Googles referer and allowing it to bypass authentication.)
    – davidgo
    Feb 25 at 23:54
  • The live test now shows the login page. Definitely not working now while it was before. I am looking for any code that was serving different pages to the crawler but have not ID'd anything yet
    – James L.
    Feb 27 at 21:46
  • Update: removed password protection and it renders fine now. Password protected sites shouldn't be crawled
    – James L.
    Oct 13 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.