5

I am finishing a new project. I have been working local for the last weeks/months. Now I want to begin the first tests on the real server and sql database. I do not want the site to be public yet and I do not want the search engines take the site into account till is ready (that will be soon but I will need a week or two) How should I do that?

I know that for a single page I should put that in the head:

<meta name="robots" content="noindex" />

For an entire site, should I put that line of code on all the pages or there is a better way?

Improve this question

4 Answers 4

Reset to default
5

Nick's answer to add robots.txt might not stop Google from index, it just says stop crawling but Google might index - official link here.

Adding meta noindex is too risky if you are working with a team that is not aware of implications of noindex tag, if it is pushed to production env. 

<meta name="robots" content="noindex">

I would recommend you to go with password protecting your website, that seems to work the best and easier to manage, with team members who are not big on SEO. And does the job of stopping access to users as well as search engines.

Improve this answer
3
  • They say they already know how to noindex pages using that meta tag. Maybe it is too hard for them to edit it into every page? +1 for your password protection suggestion though.
    – Stephen Ostermiller ♦
    Commented Aug 18, 2018 at 20:17
  • Yeah, I got that, just wanted to let @Nrc know that even if they implement it, what might happen if necessary precautions are not taken. Thanks for the upvote.
    – AnkDasCo
    Commented Aug 18, 2018 at 21:32
  • @AnkDasCo, can you please explain why it's risky to add noindex? You just presented a claim without supporting it or explaining it. I would really appreciate it cause I do need to understand why as I'm adding this to a new website.
    – Saeed Neamati
    Commented Jul 23, 2023 at 9:36
2

It's a bad idea to use robots.txt to stop the crawl on a site that you do not want indexed.

If you prevent Google from crawling your site, it will never be able to actually see your pages and, thus, will never see the noindex tag or header. So once a page is in the index and is blocked by robots.txt, Google will never see the tag telling them to remove it.

If your only concern is indexing (not, for example, a random human visitor finding it), you can have your server send a noindex header.

On Apache using .htaccess, it would be

Header set X-Robots-Tag "noindex, nofollow"

See the article on Yoast for more details. https://yoast.com/prevent-site-being-indexed/

Improve this answer
1
  • The header route is probably the best way to go. It makes it easy to blanket noindex the whole site, which is what the asker is looking for.
    – Maximillian Laumeister ♦
    Commented Jan 8, 2019 at 5:03
-1

To exclude all robots from the entire server:

User-agent: *
Disallow: /

This code will help you to get all files unindexing from server.

To exclude all robots from part of the server:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
Improve this answer
1
  • As other answers point out, using robots.txt to prevent crawling won't necessarily prevent indexing. Google does index things it can't crawl using anchor text from inbound links. I would not recommend using this answer if you want to prevent indexing and not just crawling.
    – Stephen Ostermiller ♦
    Commented Jan 8, 2019 at 11:18
-2

You can request that certain search engines not crawl your site by adding the following to your robots.txt file

User-agent: * 
Disallow: /

This will stop Google from crawling your site but your site still may be indexed. Keep in mind however that other search engines may not respect your robots.txt file at all.

You should consider adding a user/pass restriction to your site. More information on how to do this can be found here: http://www.htaccesstools.com/htaccess-authentication/

Improve this answer
5
  • 2
    adding robots.txt just not ensure not indexing, Google might still index your website. Official reference - support.google.com/webmasters/answer/6062608?hl=en
    – AnkDasCo
    Commented Aug 18, 2018 at 18:34
  • Google only indexes sites it can't crawl if it finds fairly high PageRank links to them. For a site that is under development and not likely to have any links (let alone powerful ones), robots.txt is a fine solution.
    – Stephen Ostermiller ♦
    Commented Aug 18, 2018 at 20:16
  • 3
    I would suggest to edit your answer and replace "index"/"indexing" with "crawl"/"crawling". Whether or not it makes a difference in OP’s case, robots.txt can block crawling (not indexing), noindex can prevent indexing (not crawling).
    – unor
    Commented Aug 19, 2018 at 2:29
  • The disallow requests that google not crawl nor index your site
    – Nick Duncan
    Commented Aug 19, 2018 at 4:52
  • Only crawl, but it doesn't stop indexing as mentioned in the link AnkDasCo posted.
    – Max
    Commented Aug 21, 2018 at 3:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.