I am going to create a web page, and I will do it online.
Should I choose the option: "Allow search engines to index this site." during development, or should this be opened after development ends?
Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.Sign up to join this community
Development never really ends. Every active website could have an "under construction" banner on it.
You should allow Googlebot to crawl your site as soon as you have your first piece of content that users would find useful.
Some webmasters even allow Google to crawl a "coming soon" notice that they put up as the only thing on the home page. I'm not a big fan of "coming soon", but I don't know of any good reasons that you shouldn't allow such a page to be crawled by Google.
The only time that you should prevent search spiders from crawling an entire website would be when site has information on it that you don't want to have indexed in the search engines.
Its best to
block allow while starting a new website,
If you are rebuilding an existing site then you can't do
The main reason is while you creating new site you should have some dummy content on it,
The Search Engine will crawl the site they indexed your dummy content is a bad idea.
So my suggestion is to disallow Search Engine Crawl from robots.txt until you site have proper content or launched.
Also while developing the site you can access the site without domain name like direct ip its also good practice.
Hope it make sense..