I am building a new site, but creating the content using a subdirectory on an existing site. I have never posted a link to this subdirectory anywhere, yet searching Google reveals that Google has found these pages on my site and indexed them.

Further, I updated the site with a robots.txt file that WMT reports as being read, and testing the blocked pages using the WMT test tool indicates that the pages will not be indexed. Apparently this does not remove the pages from existing results, though I submitted a request to do so.

How did Google find these pages to begin with? Is it because I used a common subdirectory name and Google guesses at common subdirectories?

  • 1
    Are you using a CMS / wiki / blog / etc. platform of some kind? Some of those ping Google whenever you add new pages. Commented Aug 27, 2013 at 23:30
  • Yes, I am using MediaWiki. I was unaware this software would do that.
    – Paul
    Commented Aug 27, 2013 at 23:31
  • Actually, AFAIK, MediaWiki normally doesn't do that. But if you added the new pages to an existing wiki that Google knew about, then it could've reached them via any number of pages (e.g. Special:Recentchanges, Special:Allpages, Special:Random...). Commented Aug 27, 2013 at 23:33
  • Although I've messed around with MediaWiki before, as far as I can recall I have never linked to anything in the /wiki/ directory.
    – Paul
    Commented Aug 27, 2013 at 23:37
  • 1
    I reviewed the links to my site and none of them link to the subdirectory.
    – Paul
    Commented Aug 27, 2013 at 23:50

3 Answers 3


Many CMS systems such as WordPress tell Google about each page that is published. WordPress calls this "update services" and uses ping-o-matic by default.

Google may also use data it gets from browsers to start crawling. If you are using a browser with a Google Toolbar (or PageRank checker), then Google gets a list of all the pages that you visit. However, Google denies that they use toolbar data for this purpose.

Google does say that a common way for "secret" URLs to be discovered is for them to link out to other sites. Those other sites then see the "secret" page in the referrer and sometimes publish a list of referrer links (a common feature of blogs).

  • I don't think WordPress could learn about pages not within WordPress and these pages were created in MediaWiki. I don't use Google Toolbar or PageRank checker.
    – Paul
    Commented Aug 31, 2013 at 21:52
  • I added another possibility and some references. Commented Sep 1, 2013 at 11:00
  • Okay, now that makes sense. Of the sites I link to, I would suspect Ars Technica of doing such as their site is a highly customized WordPress and I have one other WordPress blog I link, while the other sites are ServerFault, Wikipedia, and Ubuntu.org. Any idea where such a list would be published on a site?
    – Paul
    Commented Sep 8, 2013 at 18:28

I think when using Chrome or Google Toolbar, Googlebot will follow you while browsing your own website. So it will reach pages that you reached, regardless of links pointing to them.

About removing the content from Google index, I think that takes a while.

  • 1
    Source for this assertion? Commented Jan 13, 2017 at 20:15
  • 1
    I noticed it myself when creating php scripts and websites for the last ~12 years. Sometimes a temporary script I create would send me an email or save a log, I very often see that script being visited by bots minutes or hours after me (Alexa, Google, etc).
    – adrianTNT
    Commented Jan 14, 2017 at 23:42

Please post here the Robots.txt codes you used to block the access. Google cannot crawl website unless you have updated your robots.txt later after launching those pages because Google does crawl quickly and it has index the URL before you have actually updated your robots file. If it has happened then wait, the Google will automatically de-index your URL.

  • I didn't initially use a robots.txt file because I had no links to the pages, so I didn't think Google could find them. The robots.txt file is now: User-agent: * Disallow: /wiki/* Disallow: /piwik/*
    – Paul
    Commented Sep 8, 2013 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.