0

I have a website and I don't want this website to be visible on Google. So I deployed a robots.txt file. But the homepage of the website is still visible on Google. Here is the full code of my robots.txt copy-pasted below-

User-agent: *
Disallow: /cgi-bin/
Disallow: /private/
Disallow: /tmp/
Disallow: /_/
Disallow: /cgi-bin/
Disallow: /index.html
Disallow: /images
Disallow: /about.html
Improve this question
3
  • 2
    "... still visible on Google." - Just to add, if the homepage is already indexed, then robots.txt is not a particularly quick way of removing it.
    – MrWhite
    Commented Mar 9, 2017 at 15:37
  • 1
    Robots.txt provides instructions for spiders, but there are other ways that Google can find your site. Use noindex to specifically tell Google not to index it.
    – StephenCollins
    Commented Mar 9, 2017 at 18:17
  • @w3dk Your are right - not fast at all, however, I do not think there is a faster way. If the robots.txt file is read, it can be 24-48 hours until removal. Google does not have to fetch the page to enforce this, it is a simple query type of operation against any URL that fits the relatively simple regex(ish) pattern.
    – closetnoc
    Commented Mar 9, 2017 at 23:41

2 Answers 2

Reset to default
6

If you don't want to allowed to crawl your website on Google search result, then use this robots.txt

User-agent: *
Disallow: /

These will block all directories.

But if many website links to your website, then Google will start displaying this snippet in future.

enter image description here

Now if you don't want to index that website competely, then use noindex meta-tags/HTTP headers. That meta tags simply no index your all pages, but it is allowed to crawl. So if your main concern is about to, not visible in search result, then I highly recommended to use noindex tags.

But don't use it both, because when the site is blocked by robots.txt then Google will not going to see your meta-tags/http headers.

Improve this answer
0

Use Disallow: / to block your entire domain from being crawled by google bots.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.