Tag Info

New answers tagged


I may be somewhat overboard with my regex part, but these lines in your apache configuration will block them. Basically it scans the entire user agent identification string to see if it contains "googlebot", "bingbot" or "alexa" and if it does then direct users to the default error 403 page RewriteEngine On RewriteCond %{HTTP_USER_AGENT} ^(.*)googlebot(.*)$ ...


You can easily test in your Google Webmaster Tools account. But the answer is, no. It does not block your whole site. You block the /cgi-bin/ directory in line three. Line two essentially explicitly allows your entire site to be crawled. See the robots.txt documentation for more on this.


As tillinberlin hints at, the reason this page is appearing in the search results is because of your "robots.txt" file, however, not for the reasons given. Basically, your robots.txt file is blocking that URL from being crawled, so Google is unable to see the robots meta tag that prevents the page from being indexed. As stated in the (Google) search results ...


Short answer: robots.txt is a recommendation search engines may cherish – but they don't have to. So whatever you intend to do – don't rely on robots.txt – same applies to robots meta tags. If you really want those pages not indexed / not to be opened through search engine result pages, then you should probably add a 301 redirect or the like for everybody ...


It seems these days that a sitemap in the root named sitemap.xml will get picked up by dang near anyone in time. It all depends on how much control you want. If you do not want just anyone reading your sitemap including scrapper bots, then name it something unique and submit it to Google, Bing, and who ever else you want. Do not put it in your robots.txt ...


Instance 1 Disallow: /? This will disallow any URL's where ? follows a /. For example:- http://www.example.net/? http://www.example.net/page/? http://www.example.net/page/sub/? http://www.example.net/page/sub/?89y398th03th Instance 2 Disallow: /*?* This will disallow any URL's where ? exists in the URL after a /:- http://www.example.net/? ...


It's not supported at all since it is a not a HTTP call. It is embedded in the document and is essentially part of the HTML.

Top 50 recent answers are included