I have done google and get the two solution for disallow whole a site from
google crawl.
1:
User-agent: *
Disallow:
2:
User-agent: *
Disallow: /
Now can anyone tell me which proper code for disallow whole site from Google crawl
-
Of course the solution won't disallow just google crawl but instead all (robots-compliant) crawlers– Hagen von EitzenCommented Jul 2, 2015 at 12:26
Add a comment
|
1 Answer
To prevent your whole site from being crawled, then No. 2:
User-agent: *
Disallow: /
This blocks every URL from being crawled. The URL-path following the Disallow: directive is a prefix. If the requested URL starts with this URL-path, it will be blocked. The minimum URL path you can have is / (your home page / document root) - you can't have an empty path (as suggested in comments). When you request example.com, the browser actually requests example.com/ to make the request valid. See my other answer for more information on the trailing slash.
Disallow: by itself (without a path) actually allows everything - the complete opposite!
Reference:
http://www.robotstxt.org/robotstxt.html
-
that mean if i use no 2 solution then www.example.com will crawl and www.example.com/ does not crawl. am i right? Commented Jul 2, 2015 at 10:25
-
1No,
Disallow: /blocks everything. Or rather, the minimum request you can make is/, you can't request nothing. When you requestexample.com, the browser actually requestsexample.com/.– MrWhiteCommented Jul 2, 2015 at 10:29 -
thanks for you answer,Kindly update this comment to answer: Commented Jul 2, 2015 at 10:31
-
-
1I updated my answer and added a link for more information with regards to the trailing slash.– MrWhiteCommented Jul 2, 2015 at 10:44