1

I have done google and get the two solution for disallow whole a site from

google crawl.

1:

  User-agent: *
    Disallow:

2:

User-agent: *
Disallow: /

Now can anyone tell me which proper code for disallow whole site from Google crawl

  • Of course the solution won't disallow just google crawl but instead all (robots-compliant) crawlers – Hagen von Eitzen Jul 2 '15 at 12:26
2

To prevent your whole site from being crawled, then No. 2:

User-agent: *
Disallow: /

This blocks every URL from being crawled. The URL-path following the Disallow: directive is a prefix. If the requested URL starts with this URL-path, it will be blocked. The minimum URL path you can have is / (your home page / document root) - you can't have an empty path (as suggested in comments). When you request example.com, the browser actually requests example.com/ to make the request valid. See my other answer for more information on the trailing slash.

Disallow: by itself (without a path) actually allows everything - the complete opposite!

Reference:
http://www.robotstxt.org/robotstxt.html

  • that mean if i use no 2 solution then www.example.com will crawl and www.example.com/ does not crawl. am i right? – Amit Bera Jul 2 '15 at 10:25
  • 1
    No, Disallow: / blocks everything. Or rather, the minimum request you can make is /, you can't request nothing. When you request example.com, the browser actually requests example.com/. – MrWhite Jul 2 '15 at 10:29
  • thanks for you answer,Kindly update this comment to answer: – Amit Bera Jul 2 '15 at 10:31
  • Vote +accept.... – Amit Bera Jul 2 '15 at 10:32
  • 1
    I updated my answer and added a link for more information with regards to the trailing slash. – MrWhite Jul 2 '15 at 10:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.