Tag Info

New answers tagged


You can easily test in your Google Webmaster Tools account. But the answer is, no. It does not block your whole site. You block the /cgi-bin/ directory in line three. Line two essentially explicitly allows your entire site to be crawled. See the robots.txt documentation for more on this.


Try using http://www.google.com/analytics/, they'll provide a code spinet to insert on the head section of your website.


You really don't need to. Crawlers will find you on their own. However, if you would like to help that process a bit then include your domains in Google Webmaster Tools and Bing Webmaster Tools. Both will provide you with a number of "helpful" stats on your site as well. Also, make sure that you have an XML sitemap on your site. It will assist the crawlers ...


As tillinberlin hints at, the reason this page is appearing in the search results is because of your "robots.txt" file, however, not for the reasons given. Basically, your robots.txt file is blocking that URL from being crawled, so Google is unable to see the robots meta tag that prevents the page from being indexed. As stated in the (Google) search results ...


Short answer: robots.txt is a recommendation search engines may cherish – but they don't have to. So whatever you intend to do – don't rely on robots.txt – same applies to robots meta tags. If you really want those pages not indexed / not to be opened through search engine result pages, then you should probably add a 301 redirect or the like for everybody ...


No google show results based on users adding their webpage indexes because they are using that kind of different algorithm


Using site: is highly unreliable for counting indexed pages. It is often out of date, incorrect and actual serps is limited to sample data only. You should opt to use Google Webmaster Tools for a more accurate index count.

Top 50 recent answers are included