Skip to main content
replaced http://webmasters.stackexchange.com/ with https://webmasters.stackexchange.com/
Source Link

To prevent your whole site from being crawled, then No. 2:

User-agent: *
Disallow: /

This blocks every URL from being crawled. The URL-path following the Disallow: directive is a prefix. If the requested URL starts with this URL-path, it will be blocked. The minimum URL path you can have is / (your home page / document root) - you can't have an empty path (as suggested in comments). When you request example.com, the browser actually requests example.com/ to make the request valid. See my other answer for more information on the trailing slashmore information on the trailing slash.

Disallow: by itself (without a path) actually allows everything - the complete opposite!

Reference:
http://www.robotstxt.org/robotstxt.html

To prevent your whole site from being crawled, then No. 2:

User-agent: *
Disallow: /

This blocks every URL from being crawled. The URL-path following the Disallow: directive is a prefix. If the requested URL starts with this URL-path, it will be blocked. The minimum URL path you can have is / (your home page / document root) - you can't have an empty path (as suggested in comments). When you request example.com, the browser actually requests example.com/ to make the request valid. See my other answer for more information on the trailing slash.

Disallow: by itself (without a path) actually allows everything - the complete opposite!

Reference:
http://www.robotstxt.org/robotstxt.html

To prevent your whole site from being crawled, then No. 2:

User-agent: *
Disallow: /

This blocks every URL from being crawled. The URL-path following the Disallow: directive is a prefix. If the requested URL starts with this URL-path, it will be blocked. The minimum URL path you can have is / (your home page / document root) - you can't have an empty path (as suggested in comments). When you request example.com, the browser actually requests example.com/ to make the request valid. See my other answer for more information on the trailing slash.

Disallow: by itself (without a path) actually allows everything - the complete opposite!

Reference:
http://www.robotstxt.org/robotstxt.html

More information based on comments / added link
Source Link
MrWhite
  • 43.1k
  • 4
  • 50
  • 90

To prevent your whole site from being crawled, then No. 2:

User-agent: *
Disallow: /

This blocks every URL from being crawled. The URL-path following the Disallow: directive is a prefix. If the requested URL starts with this URL-path, it will be blocked. The minimum URL path you can have is / (your home page / document root) - you can't have an empty path (as suggested in comments). When you request example.com, the browser actually requests example.com/ to make the request valid. See my other answer for more information on the trailing slash.

Disallow: by itself (without a path) actually allows everything - the complete opposite!

Reference:
http://www.robotstxt.org/robotstxt.html

To prevent your whole site from being crawled, then No. 2:

User-agent: *
Disallow: /

Disallow: by itself (without a path) actually allows everything!

Reference:
http://www.robotstxt.org/robotstxt.html

To prevent your whole site from being crawled, then No. 2:

User-agent: *
Disallow: /

This blocks every URL from being crawled. The URL-path following the Disallow: directive is a prefix. If the requested URL starts with this URL-path, it will be blocked. The minimum URL path you can have is / (your home page / document root) - you can't have an empty path (as suggested in comments). When you request example.com, the browser actually requests example.com/ to make the request valid. See my other answer for more information on the trailing slash.

Disallow: by itself (without a path) actually allows everything - the complete opposite!

Reference:
http://www.robotstxt.org/robotstxt.html

Source Link
MrWhite
  • 43.1k
  • 4
  • 50
  • 90

To prevent your whole site from being crawled, then No. 2:

User-agent: *
Disallow: /

Disallow: by itself (without a path) actually allows everything!

Reference:
http://www.robotstxt.org/robotstxt.html