Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a robots.txt file in the root to prevent search engines indexing any of my sites. I have a social media management application embedded on my site that includes all major social media channels including G+.

I heard that once the G+ symbol is on a site Google will index sites despite having a robots.txt in the root. Is this true?

My thought behind locking out robots for a while is so I increase visitors per day first before allowing robots to crawl. Once I e.g. reach 100 visitors a day I will remove robots.txt file and will then receive a better initial page rank (in mind that PR is only updated every couple of months). Is this reasonable? How much emphasize is placed on the factor %visitor increase/day?

share|improve this question
Remember to give Google ample time to add your G+ logo. Patience is a virtue! – toughskin Nov 17 '13 at 12:43
Page rank is calculated on who links to you. It has nothing to do with amount or frequency of visitors. – Max Nov 18 '13 at 2:13

If you put a Google +1 button on your page, Googlebot will crawl the page even if the page is blocked by robots.txt. From the Google+ Webmaster FAQ:

Does +1 affect how Google crawls my site?

When you add the +1 button to a page, Google assumes that you want that page to be publicly available and visible in Google Search results. As a result, we may fetch and show that page even if it is disallowed in robots.txt.

share|improve this answer

You appear to have many misconceptions about how search engines and ranking works, so I would suggest reading through the various Google help files. Google itself has an SEO starter guide. To answer some basic points:

  • PageRank is based purely on links to your site and nothing else.
  • If your site is new then Googlebot will not be visiting very often anyway, so there is no reason to block it.
  • The number of visitors to your site (or visitor increases) is not a ranking factor at all. Google cannot possibly know how many people are visiting your site besides the ones it sends directly.
  • robots.txt prevents crawling of a page, but the URL may appear in search results under certain conditions.
  • However as Stephen points out in his answer, Google may crawl URLs with Google+ buttons, even if they are blocked by robots.txt
share|improve this answer
You have robots.txt backwards. It prevents crawling of your site but does not prevent indexing. – Stephen Ostermiller Nov 18 '13 at 11:20
@Stephen yes you are correct, thanks. – DisgruntledGoat Nov 18 '13 at 13:13

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.