How to temporarely remove a website from search engines?

Question

we are working on a new website and we forgot to disallow robots after putting the test version online, on a subdomain. now we are being already referenced by bots (google, yahoo...) and we dont want that for our test version which is not ready and finalized yet. My question is related to indexing:

• is it ok if we disallow robots at this level without having any effect on the live website (same domain) when switching from allow all to disallow all?
• can we use the 307 http status (Temporary Redirect)?
• how long doest it take spiders to refresh their db (either disallow robots or 307 redirect)?

Thanks !!

Answer 1

is it ok if we disallow robots at this level without having any effect on the live website (same domain) when switching from allow all to disallow all?

Yes. Once you stop blocking the contents of the website it will be crawled again. But I would resubmit my site to Google and/or submit a XML sitemap just to make sure they know you now have content to index.

can we use the 307 http status (Temporary Redirect)?

That would work as well.

how long doest it take spiders to refresh their db (either disallow robots or 307 redirect)?

It varies. Newer sites will find they are crawled less frequently and changes made to the website are picked up slowly. I definitely wouldn't expect it to happen quickly. You'll find in questions asked here that it can take weeks or months to finally see results.

Answer 2

If you don't want the site crawled at all, I'd suggest moving it to a "dummy" subdomain, and be sure to update the rules to NOT crawl it before putting the code there (and ideally ensure there are no links from the "live" site to the "demo" site.

e.g.

a347demo.example.com