2

I have a robots.txt file in the root to prevent search engines indexing any of my sites. I have a social media management application embedded on my site that includes all major social media channels including G+.

I heard that once the G+ symbol is on a site Google will index sites despite having a robots.txt in the root. Is this true?

My thought behind locking out robots for a while is so I increase visitors per day first before allowing robots to crawl. Once I e.g. reach 100 visitors a day I will remove robots.txt file and will then receive a better initial page rank (in mind that PR is only updated every couple of months). Is this reasonable? How much emphasize is placed on the factor %visitor increase/day?

2
  • Remember to give Google ample time to add your G+ logo. Patience is a virtue! – dvnkiss Nov 17 '13 at 12:43
  • Page rank is calculated on who links to you. It has nothing to do with amount or frequency of visitors. – Max Nov 18 '13 at 2:13
1

If you put a Google +1 button on your page, Googlebot will crawl the page even if the page is blocked by robots.txt. From the Google+ Webmaster FAQ:

Does +1 affect how Google crawls my site?

When you add the +1 button to a page, Google assumes that you want that page to be publicly available and visible in Google Search results. As a result, we may fetch and show that page even if it is disallowed in robots.txt.

1

You appear to have many misconceptions about how search engines and ranking works, so I would suggest reading through the various Google help files. Google itself has an SEO starter guide. To answer some basic points:

  • PageRank is based purely on links to your site and nothing else.
  • If your site is new then Googlebot will not be visiting very often anyway, so there is no reason to block it.
  • The number of visitors to your site (or visitor increases) is not a ranking factor at all. Google cannot possibly know how many people are visiting your site besides the ones it sends directly.
  • robots.txt prevents crawling of a page, but the URL may appear in search results under certain conditions.
  • However as Stephen points out in his answer, Google may crawl URLs with Google+ buttons, even if they are blocked by robots.txt
1
  • You have robots.txt backwards. It prevents crawling of your site but does not prevent indexing. – Stephen Ostermiller Nov 18 '13 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.