0

I have basic auth set up and it has always worked.

Suddenly Google started crawling my pages. The auth is still there (I have checked it using different browsers).

I am at a loss how it's possible.

The user/pass is dead simple to guess from the URL, does Google, by any chance, try to guess passwords?

Another guess is I at some point entered the password somewhere in Google admin. I don't even know that's possible, but does anyone have any idea if it can be done? I have wasted my whole day trying to figure this one out!

  • 2
    ...does google, by any chance, try to guess passwords? No. Your system is letting the googlebot in. That is not a bad thing unless you do not want these pages indexed. Check you sitemap for URLs that are protected by a login. Perhaps use your robots.txt file to block indexing. How people give googlebot access to secure pages is not something I know about so I cannot tell you what to look for. If you are using a CMS, then there may be a default behavior to allow Google access. – closetnoc Jan 4 '17 at 1:10
  • googlebot doesnt accept cookies. if your auth works cookie-based - it wouldn't lock out googlebot. check your auth with browser, which doesn't accept cookies – Evgeniy Jan 5 '17 at 8:34
4

Google has a feature in webmaster tools where you can add login information if you want Google to crawl content behind a user login form. If you have provided Google this information in the past and have not changed the login information since then Google will have access to the content accessible to the login information you provided to it. Google does not however try to "guess" login information even if this can be parsed from the URL of the page.

Google can also try to crawl a page that is protected by a user login if the page exists in your sitemap file or has been linked to from another website or a link on your site that Google has access to. In this case Google will try to crawl the page and will detect that it is protected by HTTP basic authentication and so won't list it in the index but it a crawl attempt will still have been attempted.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.