Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 2 down vote favorite
1

We've recently bought couple of domains but the content of the domains is still under development.

Which content should we upload to the domains to tell search engines "There is currently nothing, but check in the future" and don't get crawling/ranking penalty for the future content?

share|improve this question

3 Answers

up vote 0 down vote

Use robots.txt to disallow as RandomBen suggests. Search engines crawlers may not come back very often if it finds all your content is disallowed.

However, once your site is up and ready, change your robots.txt and resubmit to Google (from webmaster tools), Bing, etc.

There should be no penalty and when you resubmit your new robots.txt the search engines will begin to crawl and index your site properly.

share|improve this answer
up vote 0 down vote

I see few possible approaches (in descending order):

  1. Just restrict access by IP: only allow certain IPs or subnets (in case your IP is dynamic) to access the site and block all other requests - the most reliable approach.

  2. Display empty page or some "Coming Soon" text (or just nice big image and no text at all) and use subdomain to access your current live site (for example: beta.example.com). Unfortunately it is not that reliable as blocking by IP as search engine still may detect the subdomain or discover a link to it (Google Desktop search, links in emails sent to @gmail.com address, someone publishes a link somewhere and it was indexed etc).

  3. Use robots.txt -- this will work fine with bots that respect it, but offer no "protection" otherwise. If your only concern is Google/Bing etc then it should work fine.

share|improve this answer
up vote 4 down vote

I would add a robots.txt file to all of these sites with the code below in it if you are worried about Google attempting to crawl your site:

User-agent: *
Disallow: /

Remember, Google needs to know your site exists before they will even scan it. So if you haven't created any links to the site or a Google Webmaster/Google Analytics account for the site yet, Google probably won't know about it. In addition, Google should not penalize you for having an empty site once you do have a site. That is a normal state for many domains before they are setup.

share|improve this answer
I believe Google will not penalize the site, but maybe some other search engines could. One of factors which search engines watch is how often is page changed, how old is domain etc. The robots.txt looks like a good solution, thanks. – Petr Peller Aug 23 '11 at 13:14
I am only concerned about that some crawlers may give up trying to crawl the page when they see the robots.txt disallowing all pages too many times. – Petr Peller Aug 23 '11 at 13:19
I cannot speak to crawlers I don't know but Google and Bing both allow you to add your site through their webmaster portal and request a rescan, so I wouldn't be worried. They look for how often info is updated and increase their scans the more often you update your site. – RandomBen Aug 23 '11 at 15:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.