Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I am building a new site, but creating the content using a subdirectory on an existing site. I have never posted a link to this subdirectory anywhere, yet searching Google reveals that Google has found these pages on my site and indexed them.

Further, I updated the site with a robots.txt file that WMT reports as being read, and testing the blocked pages using the WMT test tool indicates that the pages will not be indexed. Apparently this does not remove the pages from existing results, though I submitted a request to do so.

How did Google find these pages to begin with? Is it because I used a common subdirectory name and Google guesses at common subdirectories?

share|improve this question
    
Are you using a CMS / wiki / blog / etc. platform of some kind? Some of those ping Google whenever you add new pages. –  Ilmari Karonen Aug 27 '13 at 23:30
    
Yes, I am using MediaWiki. I was unaware this software would do that. –  Paul Aug 27 '13 at 23:31
    
Actually, AFAIK, MediaWiki normally doesn't do that. But if you added the new pages to an existing wiki that Google knew about, then it could've reached them via any number of pages (e.g. Special:Recentchanges, Special:Allpages, Special:Random...). –  Ilmari Karonen Aug 27 '13 at 23:33
    
Although I've messed around with MediaWiki before, as far as I can recall I have never linked to anything in the /wiki/ directory. –  Paul Aug 27 '13 at 23:37
1  
I reviewed the links to my site and none of them link to the subdirectory. –  Paul Aug 27 '13 at 23:50
show 3 more comments

3 Answers 3

Many CMS systems such as WordPress tell Google about each page that is published. WordPress calls this "update services" and uses ping-o-matic by default.

Google may also use data it gets from browsers to start crawling. If you are using a browser with a Google Toolbar (or PageRank checker), then Google gets a list of all the pages that you visit. However, Google denies that they use toolbar data for this purpose.

Google does say that a common way for "secret" URLs to be discovered is for them to link out to other sites. Those other sites then see the "secret" page in the referrer and sometimes publish a list of referrer links (a common feature of blogs).

share|improve this answer
    
I don't think WordPress could learn about pages not within WordPress and these pages were created in MediaWiki. I don't use Google Toolbar or PageRank checker. –  Paul Aug 31 '13 at 21:52
    
I added another possibility and some references. –  Stephen Ostermiller Sep 1 '13 at 11:00
    
Okay, now that makes sense. Of the sites I link to, I would suspect Ars Technica of doing such as their site is a highly customized WordPress and I have one other WordPress blog I link, while the other sites are ServerFault, Wikipedia, and Ubuntu.org. Any idea where such a list would be published on a site? –  Paul Sep 8 '13 at 18:28
add comment

I think when using Chrome or Google Toolbar, Googlebot will follow you while browsing your own website. So it will reach pages that you reached, regardless of links pointing to them.

About removing the content from Google index, I think that takes a while.

share|improve this answer
add comment

Please post here the Robots.txt codes you used to block the access. Google cannot crawl website unless you have updated your robots.txt later after launching those pages because Google does crawl quickly and it has index the URL before you have actually updated your robots file. If it has happened then wait, the Google will automatically de-index your URL.

share|improve this answer
    
I didn't initially use a robots.txt file because I had no links to the pages, so I didn't think Google could find them. The robots.txt file is now: User-agent: * Disallow: /wiki/* Disallow: /piwik/* –  Paul Sep 8 '13 at 18:29
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.