Tag Info

Hot answers tagged

22

Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt. Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.


13

Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure If your site is password protected properly, there shouldn't be any need to block particular user agents :) x


13

Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic. Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...


11

Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless ...


9

You can add a dollar sign to the end of the string which means it will only match exactly that entry: # Files User-agent: * Disallow: /mage$ This will only block the mage file if it come straight after the root domain: www.example.com/mage If there are any other preceding directories, you must add these o the entry. So to block the file located ...


7

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


5

If you are using disallow:/ with user-agent: * then only entire site is blocked with robots.txt. Otherwise only particular user-agent will get blocked with your sites. Blocking user-agents: The Disallow line lists the pages you want to block. You can list a specific URL or a pattern. The entry should begin with a forward slash (/). To block the entire ...


5

Here is a robots.txt file that will allow Google, Bing, and Yahoo to crawl the site while disallowing all other crawling: User-Agent: * Disallow: / User-Agent: googlebot Disallow: User-Agent: bingbot Disallow: User-agent: slurp Disallow: Some crawlers ignore robots.txt entirely and crawl whatever they feel like. Some crawlers impersonate Googlebot or ...


5

Whether or not crawlers honor your robots.txt is entirely an on-your-honor based system. Nothing you put in that file is going to prevent a "fake" crawler from doing anything. With regards to User-agent:, that value is completely voluntary as well. You can instruct your browser, or any other HTTP client to send whatever value you want for that header.


5

The default behaviour of robots if there is no robots.txt and robots meta tag is to follow links on the page and index it. That's why <meta name="robots" content="index, follow" /> is not necessary if you want the default behaviour of robots for your page.


5

Most bots don't accept cookies (including Googlebot), however, some bots do. You send a Set-Cookie header in the response, but the bot does not send back a Cookie header in subsequent requests - so the cookie is effectively lost. Whether Google is monitoring whether the site is setting cookies, we don't know for sure, but I'd wager they probably are. ...


5

wget has legitimate uses, yes, but it's also quite useful for Web scraping. However, I don't think you should try to block it (or any other agent) by using the user agent string. wget respects, by default, your robots.txt file. It's true that a scraper can just switch that option off, but guess what -- it's just as easy to use --user-agent ...


5

I wouldn't count on all spiders being able to follow a redirect to get to a robots.txt file. See: Does Google respect a redirect header for robots.txt to a different file name? Assuming you are hosted on an Apache server, you could use mod_rewrite from your .htaccess file to to serve the correct file for the correct domain: RewriteEngine On RewriteCond ...


5

Google only cares that they see the same thing your users see. If you're only serving up different content to these bad crawlers, and your users are getting your normal content and Google is getting that same content, you're ok.


4

I think this help document from Google should be solving my problem: Change the crawl rate: On the Webmaster Tools Home page, click the site you want. Click the gear icon , then click Site Settings. In the Crawl rate section, select the option you want. The new crawl rate will be valid for 90 days.


4

Bingbot is not "crashing". The "Fetch as Bingbot" tool within Bing's Webmaster Tools simply does not follow redirects: From bing webmaster help: WHAT DOES "REDIRECTION LIMIT REACHED MEAN"? Unlike the SEO Analyzer tool, Fetch as Bingbot does not follow redirects. Instead it will let you know that the page resulted in a redirect and shows you the HTTP ...


4

Thanks! On further investigation, I also found the hint to add <meta name="robots" content="noimageindex"> to the header of the web page - it is supposed to prevent the page used as the referring page according to a few sources. I'll see if that's sufficient and update my post if necessary.


4

To prevent the image being indexed via the site you are hotlinking from, instead of linking (hotlinking) directly to the source image you could perhaps call a script which reads and serves the appropriate image from the source site together with an X-Robots-Tag: noindex HTTP response header. <img src="/get-image.php?file=my-image-hosted-elsewhere.jpg" ...


4

In the eyes of Google, a PDF is just another web page – a web page that offers a prime opportunity to boost your content ahead of your competitors and vice versa. The reason I say is that Google ranks PDF files in the SERPs. It is sure that it crawls the PDF files. If PDF content is fresh and relevant, it will increase your website reputation. It always ...


4

There are a few possible reasons why a bot would try to visit a removed page: The bot followed a link to that page from another website. Bots frequently omit referrer so it is difficult to tell if this is the case. Given that the bot in question has "backlink checker" as part of its tagline, this seems a likely cause. The bot had visited the page while it ...


4

To block all "blank" User-Agents or User-Agents consisting of a hyphen, you could use the following in your .htaccess RewriteCond %{HTTP_USER_AGENT} ^-?$ RewriteRule ^ - [F]


4

I have seen these people before and they are just what you describe. In my database, I see that they do read robots.txt, but they do not offer a bot name to block accesses to your site. This site fits my definition of a bad bot (unwanted / unappreciated). There is plenty of evidence of this to be found with a simple Google search. When in doubt, just block ...


3

Even Google will occasionally include a site in its search index, despite the fact that the site disallows all crawling with robots.txt. Google views robots.txt rules as preventing crawling, but not necessarily inclusion in the search index. When Googlebot can't crawl a site, but the site has lots of external links, Google may include pages from the site ...


3

Development never really ends. Every active website could have an "under construction" banner on it. You should allow Googlebot to crawl your site as soon as you have your first piece of content that users would find useful. Some webmasters even allow Google to crawl a "coming soon" notice that they put up as the only thing on the home page. I'm not ...


3

The current GoogleBot Smartphone agent, as tested with the 'Fetch as Google' Tool is essentially a fake iPhone using a headless Webkit Safari 6.0 Engine, running on a Linux x86_64 desktop machine. The default non-responsive viewport width is that of an iPhone at 980px. With a viewport <meta name="viewport" content="width=device-width, ...


3

Google Webmaster Tools does have the ability to tell Googlebot to ignore parameters. You can tell Googlebot to ignore the parameter "MobileOptOut" anywhere it appears on your site. To do so: Navigate in Webmaster Tools to Crawl -> URL Parameters. Click the Configure URL parameters » link. Click the Add Parameter button. Put in the name: "MobileOptOut". ...


3

Stephen Ostermiller’s answer explains how to configure it in Google Webmaster Tools (which answers your question). If you care about other search engines, too, or in addition, you could use the Canonical link type. You could specify the canonical link in a HTTP header, or you could specify it in the HTML: On ...


3

I think that you may be interested in this article: Design AJAX-powered sites for accessibility. It's a very basic article, but it should give you some pointers about what to do and what not to do. AJAX should be used to improve the experience on specific pages and elements of the site, but not to build a whole site. For the user perspective, it's a bad ...


3

While this should work, it has a few potential drawbacks: Every crawler has to do two HTTP requests: one to discover the redirect, and another one to actually fetch the file. Some crawlers might not handle the 301 response for robots.txt correctly; there's nothing in the original robots.txt specification that says anything about redirects, so presumably ...


3

In order to ensure that Google has crawled and indexed your content, you can just do a search with the keyword you'd like to be indexed. E.g., you've a heading as follows: <h1>The h1 heading on my site</h1> Try to search with Google the 'The <h1> heading on my site' phrase. If the SERP will contain the keyword and a link to your ...



Only top voted, non community-wiki answers of a minimum length are eligible