We are working on a new website and we forgot to disallow robots after putting the test version online, on a subdomain. Now we are already being crawled and indexed by bots (Google, Yahoo, and others) and we don't want that for our test version which is not ready and finalized yet.
My question is related to indexing:
- Is it OK if we disallow robots at this level without having any effect on the live website (same domain) when switching from allow all to disallow all?
- Can we use the 307 HTTP status (Temporary Redirect)?
- How long doest it take spiders to refresh their db (either disallow robots or 307 redirect)?