1

One of my web page with sensitive information was indexed by BING search and which in turn showing in the search results. Now that i solved by Robots meta tag in the header. However i would like to know the Root from which BING got this link? To be specific -> Where did web crawler identified my website link.

Is there any way i could find that?

  • A secret URL is not sufficient for sensitive information. Users should have to log in to view it. One way that URLs leak it through their links. Links send a referrer with that URL. The linked site sometimes publishes a list of referring URLs. It is also possible that users' browsers or plugins report URLs to search engines. – Stephen Ostermiller Jun 11 '17 at 10:26
1

The answer to that would be to look at your website structure.

Let's say if a website is built in a custom way then it's very easy to identify as every single page code you can investigate easily.

On the other side if you got a third party system then you may not realise one of your pages may have the link which you may not be aware of and Bing might have crawled it. Also, worth checking header of the pages, meaning Bing might have got a signal from the script which may not be visible in general.

I would say there are so many third party tool which you can use to find your entire internal link pattern and you may get the lead from there.

Lastly, any private info should be well planned to ensure either they are password protected or blocked using robots.txt

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.