Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We are working on a new website and we forgot to disallow robots after putting the test version online, on a subdomain. Now we are already being crawled and indexed by bots (Google, Yahoo, and others) and we don't want that for our test version which is not ready and finalized yet.

My question is related to indexing:

  • Is it OK if we disallow robots at this level without having any effect on the live website (same domain) when switching from allow all to disallow all?
  • Can we use the 307 HTTP status (Temporary Redirect)?
  • How long doest it take spiders to refresh their db (either disallow robots or 307 redirect)?
share|improve this question

2 Answers 2

is it ok if we disallow robots at this level without having any effect on the live website (same domain) when switching from allow all to disallow all?

Yes. Once you stop blocking the contents of the website it will be crawled again. But I would resubmit my site to Google and/or submit a XML sitemap just to make sure they know you now have content to index.

can we use the 307 http status (Temporary Redirect)?

That would work as well.

how long doest it take spiders to refresh their db (either disallow robots or 307 redirect)?

It varies. Newer sites will find they are crawled less frequently and changes made to the website are picked up slowly. I definitely wouldn't expect it to happen quickly. You'll find in questions asked here that it can take weeks or months to finally see results.

share|improve this answer
    
thanks John !!! –  user8849 Jul 7 '11 at 15:56

If you don't want the site crawled at all, I'd suggest moving it to a "dummy" subdomain, and be sure to update the rules to NOT crawl it before putting the code there (and ideally ensure there are no links from the "live" site to the "demo" site.

e.g.

a347demo.example.com
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.