Tag Info

Hot answers tagged


To prevent your whole site from being crawled, then No. 2: User-agent: * Disallow: / This blocks every URL from being crawled. The URL-path following the Disallow: directive is a prefix. If the requested URL starts with this URL-path, it will be blocked. The minimum URL path you can have is / (your home page / document root) - you can't have an empty path ...


There are few steps through which you can increase the rate Google crawl: Update your website content on a regular basis Create a sitemap Do not use duplicate content Reduce your site loading time Prohibit access to unwanted page to Google via robots.txt Optimize and monitor Google crawl rate via Google webmaster tools Use pinging service Submit your site ...

Only top voted, non community-wiki answers of a minimum length are eligible