4

I am finishing a new project. I have been working local for the last weeks/months. Now I want to begin the first tests on the real server and sql database. I do not want the site to be public yet and I do not want the search engines take the site into account till is ready (that will be soon but I will need a week or two) How should I do that?

I know that for a single page I should put that in the head:

<meta name="robots" content="noindex" />

For an entire site, should I put that line of code on all the pages or there is a better way?

-2

You can request that certain search engines not crawl your site by adding the following to your robots.txt file

User-agent: * 
Disallow: /

This will stop Google from crawling your site but your site still may be indexed. Keep in mind however that other search engines may not respect your robots.txt file at all.

You should consider adding a user/pass restriction to your site. More information on how to do this can be found here: http://www.htaccesstools.com/htaccess-authentication/

  • 2
    adding robots.txt just not ensure not indexing, Google might still index your website. Official reference - support.google.com/webmasters/answer/6062608?hl=en – AnkDasCo Aug 18 '18 at 18:34
  • Google only indexes sites it can't crawl if it finds fairly high PageRank links to them. For a site that is under development and not likely to have any links (let alone powerful ones), robots.txt is a fine solution. – Stephen Ostermiller Aug 18 '18 at 20:16
  • 3
    I would suggest to edit your answer and replace "index"/"indexing" with "crawl"/"crawling". Whether or not it makes a difference in OP’s case, robots.txt can block crawling (not indexing), noindex can prevent indexing (not crawling). – unor Aug 19 '18 at 2:29
  • The disallow requests that google not crawl nor index your site – Nick Duncan Aug 19 '18 at 4:52
  • Only crawl, but it doesn't stop indexing as mentioned in the link AnkDasCo posted. – Max Aug 21 '18 at 3:17
5

Nick's answer to add robots.txt might not stop Google from index, it just says stop crawling but Google might index - official link here.

Adding meta noindex is too risky if you are working with a team that is not aware of implications of noindex tag, if it is pushed to production env.

<meta name="robots" content="noindex">

I would recommend you to go with password protecting your website, that seems to work the best and easier to manage, with team members who are not big on SEO. And does the job of stopping access to users as well as search engines.

  • They say they already know how to noindex pages using that meta tag. Maybe it is too hard for them to edit it into every page? +1 for your password protection suggestion though. – Stephen Ostermiller Aug 18 '18 at 20:17
  • Yeah, I got that, just wanted to let @Nrc know that even if they implement it, what might happen if necessary precautions are not taken. Thanks for the upvote. – AnkDasCo Aug 18 '18 at 21:32
1

It's a bad idea to use robots.txt to stop the crawl on a site that you do not want indexed.

If you prevent Google from crawling your site, it will never be able to actually see your pages and, thus, will never see the noindex tag or header. So once a page is in the index and is blocked by robots.txt, Google will never see the tag telling them to remove it.

If your only concern is indexing (not, for example, a random human visitor finding it), you can have your server send a noindex header.

On Apache using .htaccess, it would be

Header set X-Robots-Tag "noindex, nofollow"

See the article on Yoast for more details. https://yoast.com/prevent-site-being-indexed/

  • The header route is probably the best way to go. It makes it easy to blanket noindex the whole site, which is what the asker is looking for. – Maximillian Laumeister Jan 8 at 5:03
-1

To exclude all robots from the entire server:

User-agent: *
Disallow: /

This code will help you to get all files unindexing from server.

To exclude all robots from part of the server:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
  • As other answers point out, using robots.txt to prevent crawling won't necessarily prevent indexing. Google does index things it can't crawl using anchor text from inbound links. I would not recommend using this answer if you want to prevent indexing and not just crawling. – Stephen Ostermiller Jan 8 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.