Hot answers tagged web-crawlers
To prevent your whole site from being crawled, then No. 2: User-agent: * Disallow: / This blocks every URL from being crawled. The URL-path following the Disallow: directive is a prefix. If the requested URL starts with this URL-path, it will be blocked. The minimum URL path you can have is / (your home page / document root) - you can't have an empty path ...
Only top voted, non community-wiki answers of a minimum length are eligible