4

Suppose I have a "hidden" page on a website, but I don't explicitly tell bots to ignore it (no noindex tag, not disallowed in robots.txt, etc). It's not linked to anywhere, doesn't appear on the sitemap.xml, and no tracking (ie Google Analytics) exist on the page.

Is visiting the page through the URL using Google Chrome enough for it to be crawled and appear as a search result on Google? What about other browsers managed by companies that also run a search engine (ie Edge/Bing)?

4

No. Google would not take the right to do that (I don't think any browser has done such). Also many "hidden" pages would anyway not be accessible; i.e. I may have many pages on my computers that are 100% private; also many websites have pages that are not accessible unless you first create an account and log in.

However, would Google be able to eventually find the page? Yes.

  1. If you have links on that page to lead to other public pages and you allow for the Referer¹, then your page URL is now out in the wild. Are referrer URL made public? Pretty rarely, but I've seen such happen in the past, so it can happen.
  2. If you tell enough people where the page is, eventually someone will make that URL public. i.e. post the URL on a forum, send it to someone else via email and that email becomes public, etc.
  3. There are also bots that will try, at least seemingly, random paths against websites. They could inadvertently discover your page that way. Those bots may then create a page on their owner's website with a link to your hidden page. I've seen such strange websites which attempt to grade your page in what at least to me looks like totally random algorithms.

You also mentioned Google Analytics. It would be interested to test (verify) that they do not share the URLs they get in Google Analytics with Google Search. They also are not supposed to do that either.

However, there is the Alexa toolbar. This one's from Amazon.com (I'm sure there are others similar toolbars). It adds a script to your browser which essentially sends all the URLs that you access to Alexa. Probably a bad idea. So such exists, but it's not in the browsers by default.

¹ the spelling of "Referer" is wrong in the HTTP headers and it was never fixed. Computers don't care much.

| improve this answer | |
  • 1
    Thanks for such an in-depth answer! For context, we are building a "microsite" as part of another larger site and were unsure if it was worthwhile to "hide" all of the pages while content is being entered and then un-hide them later. Google being able to eventually find it isn't that big of a concern. – Quangdao Nguyen Jan 6 at 1:07
  • @QuangdaoNguyen At the same time, the sooner Google sees the page, the sooner it gets in the index (because slower sites new pages do not get indexed right away...) but I understand that if you're doing heavy changes to those pages for a while, it can be better to avoid incorrect indexing (because Google won't refresh their index instantly once they found your pages for the first time). – Alexis Wilke Jan 6 at 17:58
  • 1
    When doing any development work, the correct approach is to add either "noindex, nofollow" or disallow access via robots.txt, or set a password. You don't won't SEs discovering those pages before they are ready for the "public". Widely used term for pages that are not accessible from anywhere else on the website is "orphan pages". – Mnea Jan 7 at 20:07
  • 2
    Don't rely on search engines not spotting a page to keep it private - be explicit about whether you want it indexed or not. Sometimes links to pages show up surprisingly quickly; people forward emails to mailing lists, plugins share "related pages", some sites list referrers publicly, sometimes you also accidentally include it yourself (or in a sitemap that's generated automatically). Give clear signals (noindex or not), if you want a specific action. – John Mueller Jan 8 at 8:06
  • I agree, and past (March 2017) tests appear to have shown this to be the case. However, I do wonder the extent of a more recent option in Chrome settings (disabled by default): "Make searches and browsing better / Sends URLs of pages you visit to Google". (?) – DocRoot Jan 17 at 23:58
0

I made two server programs. The program records the IP address of the site visited. They do not publish the URL in links or sitemaps. The first was Chrome, which accessed the URL directly. The other was accessed by a Microsoft Edge. As a result, programs accessed in Chrome were indexed in Google(ip:64.233.172.186)

whois(64.233.172.186)info:
NetRange:       64.233.160.0 - 64.233.191.255
CIDR:           64.233.160.0/19 
NetName:        GOOGLE
NetHandle:      NET-64-233-160-0-1
Parent:         NET64 (NET-64-0-0-0-0)
NetType:        Direct Allocation
OriginAS:       
Organization:   Google LLC (GOGL)
RegDate:        2003-08-18
Updated:        2012-02-24
| improve this answer | |
  • When you say "indexed in Google" do you mean "crawled by Googlebot"? Just because Googlebot crawls something, it doesn't mean that Google indexes it. Did you also check to see whether or not it could be found in Google's search results? – Stephen Ostermiller Apr 22 at 10:28
  • Did the pages use any third party scripts, images, or CSS? Did the pages link to any third party sites? – Stephen Ostermiller Apr 22 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.