I am going to create a web page, and I will do it online.

Should I choose the option: "Allow search engines to index this site." during development, or should this be opened after development ends?

  • 1
    Ideally, you wouldn't have a development site on the live web. But that aside, I'd probably leave the option on -- you won't get much traffic until the website has content anyway.
    – Brendon
    Dec 4 '13 at 19:49

Development never really ends. Every active website could have an "under construction" banner on it.

You should allow Googlebot to crawl your site as soon as you have your first piece of content that users would find useful.

Some webmasters even allow Google to crawl a "coming soon" notice that they put up as the only thing on the home page. I'm not a big fan of "coming soon", but I don't know of any good reasons that you shouldn't allow such a page to be crawled by Google.

The only time that you should prevent search spiders from crawling an entire website would be when site has information on it that you don't want to have indexed in the search engines.


Its best to block allow while starting a new website,

If you are rebuilding an existing site then you can't do disallow options, The main reason is while you creating new site you should have some dummy content on it, The Search Engine will crawl the site they indexed your dummy content is a bad idea.

So my suggestion is to disallow Search Engine Crawl from robots.txt until you site have proper content or launched.

Also while developing the site you can access the site without domain name like direct ip its also good practice.

Hope it make sense..


It really depends on whether you care if your site is viewed when "under construction". Usually it's best to wait until after the site is complete to index.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.