1

I have noticed that bing/msnbot tries to index pages that were only ever linked to in one single email.

My guess is, that some kind of browser toolbar is responsible for this.

Apart from robots.txt, is there something I can or should do to prevent search engines from indexing private pages, i.e. pages that could only ever be discovered by sniffing "private" content like emails or password protected pages my users browse to?

2
  • in case, this is interesting to somebody: around 10% of "private" links seem to be leaked to msn. i have not yet seen google access such pages.
    – user76725
    Commented Apr 5, 2017 at 19:29
  • You can use a noindex tag on the page. Here is the Google page on this: support.google.com/webmasters/answer/93710?hl=en Bing should respect this too. Cheers!!
    – closetnoc
    Commented Apr 5, 2017 at 21:33

1 Answer 1

1

If you specifically want to prevent a page/URL from being indexed then include a noindex robots meta tag on HTML responses (or equivalent X-Robots-Tag HTTP response header).

Do not block the page with robots.txt. The robots.txt file primarily prevents the page from being crawled (which has the added side-effect that it probably doesn't get indexed). However, pages blocked by robots.txt can still be indexed if they are linked to (a link-only result appears in the SERPs). And blocking crawling with robots.txt prevents the search engine bots from seeing the robots noindex meta tag. Including the URL in robots.txt can also be an advert to "bad" bots.

However, if the page contains sensitive information or must only be seen by the intended recipient of the email then it should be blocked by some kind of authentication.

1
  • Thanks for the explanation! Although, I must say: if I can't trust the "good" guys to not leak information from (encrypted) private emails (that are not even read on a webmail account)… Authentication is not always an option (e.g. in surveys).
    – user76725
    Commented Apr 6, 2017 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.