0

I have read this which claims that the listed search engines are not able to show the unlinked doc&XML in its results. My questions are

Have any one else experimented like this?
This information seems old, does these engines are learned to crawl them too?

Any updates on these. Thank you.

-PS.

I have seen other duplicate questions which are not answering my question.

1) My question is about any experiments have been carried after 2008. 2) the question quoted for duplicate doesn't provide any reference to the questions asked by that questions OP. 3) I have mentioned that I have read them.and not answering my questions

  • 1) My question is about any experiments have been carried after 2008. 2) the question quoted for duplicate doesn't provide any reference to the questions asked by that questions OP. 3) I have mentioned that I have read them.and not answering my questions. I feel these are enough to say that this is not duplicate!. Thank you for understanding. – ganesh Oct 19 '15 at 16:54
2

Search engines follow links and read sitemaps to do their work. If the pages are not linked to and the sitemap does not contain the URL, then Yes- search engines are unable to find the pages.

However, if anyone links to the pages, then that is a different matter. As well, social media and other references can be used to find the pages.

  • "and other references" - And that's the thing, I think Google is getting better and better (read: more cunning) in discovering pages. Links in email (gmail), Google Web History, etc. – MrWhite Oct 19 '15 at 16:37
  • @w3d You are absolutely right! There is a reason why Google created GMail and some of the other properties- they are very forward thinking and sneaky!! That is why they are #1 afterall. – closetnoc Oct 19 '15 at 16:39
  • Yes!.I understand now!.. But some of my friends were saying that, Unless I explicitly disallow the URL, the bot will find it. Have anybody else experimented it?..I don't have one to test on my own.. Thanks for your replies.. not only google..all of them – ganesh Oct 19 '15 at 16:42
  • They are right in a way. Google is very good at ferreting out new pages to index. You can block the page specifically using the robots.txt file. Web users who discover your page often will link to a page or share it eventually. No experimentation needed on this side. We have all seen how fast a page without a link can be discovered. In that, your friends are right. You should disallow the page if you do not want it indexed. – closetnoc Oct 19 '15 at 16:46

Not the answer you're looking for? Browse other questions tagged or ask your own question.