Hot answers tagged

24

Do crawlers behave differently in these two cases? A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling. You might however receive lots of 404 errors in your server logs when crawlers request the robots.txt file, as indicated in this question here. So, is it safe to just delete an empty ...


21

I'm the primary designer and author of a fairly large-scale web crawler (see http://metadatalabs.com/mlbot). What you're asking touches on a topic that's very important to us--perhaps the most important part of running a crawler: that of politeness. First: the reason for the "Mozilla" thing is to tell the site what your browser capabilities are. If your ...


21

Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt. Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.


21

The last record (started by User-agent: *) will be followed by all polite bots that don’t identify themselves as "googlebot", "google", "bingbot" or "bing". And yes, it means that they are not allowed to crawl anything. You might want to omit the * in /bedven/bedrijf/*. In the original robots.txt specification, * has no special meaning, it’s just a ...


17

Apparently it does, some, according to Matt Cutts: "For a while, we were scanning within JavaScript, and we were looking for links. Google has gotten smarter about JavaScript and can execute some JavaScript. I wouldn't say that we execute all JavaScript, so there are some conditions in which we don't execute JavaScript. Certainly there are some common, ...


14

I've used Xenu's Link Sleuth. It works pretty well, just be sure not to DOS yourself!


14

I'm glad you know there isn't going to be a bulletproof way to accomplish this. That means your outlook is at least realistic. Since JavaScript is not an option I would say you're left with: Check the user-agent for the word "bot" in it. That will catch most of them. Compile a list of known bots and filter them based on some kind of unique identifier, ...


13

Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure If your site is password protected properly, there shouldn't be any need to block particular user agents :) x


13

Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic. Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...


12

You could do several things including: Putting a fake field that only bots will see. Then if that field is submitted with the rest of the form you can ignore it (and ban them if desired). You can also trap bad bots who follow a hidden link. Use a CAPTCHA like reCAPTCHA Use a field that requires the user to answer a question like what is 5 + 3. Any human ...


12

This is Vincent from Bing Webmaster Tools. This was a Bingbot checking for an XML sitemap generated by the Bing Sitemap Plugin (Beta) for IIS and Apache - see the following for more details: Bing Sitemap Plugin (Beta) - Bing Webmaster Tools Help & How-To Center Building Sitemaps Manually?… Stop Until You Read This! - Bing Webmaster Blog You shouldn'...


12

Several other search engines (Bing, Yandex, etc.) still use the _escaped_fragment_ system. They're not going to stop using it overnight just because Google has. Thus, if you care about your site being indexable by search engines other than Google, you may want to still support this scheme. Certainly, if you already have set up support for ...


11

Yes and no, he's conflating two different things. In late 90's/early 2000's, when designers needed to use fonts for design reasons, they embedded graphics files in HTML pages. Search engines cannot understand these, nor can the screen readers used by blind and partially sighted people. This is because the only HTML code in the page for a menu link, for ...


11

We now utilize a crawling method of adding new content to our search index. The subdomain you refer to was just retired over the weekend, and we'll be updating that FAQ page very shortly. Best regards, Eric from Ask.com


11

Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless ...


11

You can make any file dynamic. The best way to do so is not through redirects, but through rewrite rules. RewriteRule ^robots\.txt$ /robots.php [L] That way, you power it with a dynamic script, but the URL doesn't change. Most crawlers (including Googlebot) will follow redirects for robots.txt, but some crawlers will get confused if you introduce ...


11

No, web crawlers will not read or obey a robots.txt file in a subdirectory. As described on the quasi-official robotstxt.org site: Where to put it The short answer: in the top-level directory of your web server. or on Google's help pages (emphasis mine): A robots.txt file is a file at the root of your site that indicates those parts of your ...


10

I would see no reason to unblock Yahoo's search bot. There are 2 ways Yahoo could be using the Bing Engine. They could integrate it with their own and set a hybrid crawler loose on the net or they could just be sending the queries into the Bing Engine and spitting out the results they get back. From the document Joel linked to there is another document ...


10

You do not need one as not having one is interpreted as meaning you want to have all of your content crawled. But I recommend using a blank one just to prevent the accumulation of unnecessary 404 errors in your stats.


10

There are at least 3 ways: Links to your site. Using Google Webmaster Tools (now called Search Console) Registrar dumps, triggers, and other options. Google will find many new sites quickly from some registrars. For example, Google found one domain name I registered using GoDaddy, indexed it, and began sending search results within 20 minutes of ...


10

It's not ARPA (Advanced Research Project Agency). That's just DNS at work. Here is a write-up on The in-addr.arpa Domain.


9

Yes, if the site requires authentication then robots will not be able to crawl it. You can also specify a robots.txt file so that the entire site is disallowed


9

Can they see it? Yes. Can they find it? Not without help. Web crawlers typically find pages to crawl by following links to them on other pages. Some crawlers (e.g. search engine crawlers) will also crawl pages listed in special XML files. So if there is no link to page on your website or any other website then that page will not be crawled (pages that ...


9

You can add a dollar sign to the end of the string which means it will only match exactly that entry: # Files User-agent: * Disallow: /mage$ This will only block the mage file if it come straight after the root domain: www.example.com/mage If there are any other preceding directories, you must add these o the entry. So to block the file located ...


9

Google expects differences between mobile and desktop sites. Even major differences, including differences in link structure, are not a problem. Google crawls the web with different Googlebot user agents for mobile. As long as your server shows that version of Googlebot the same thing that your actual mobile users see, you don't have any penalty risk. ...


9

No. There's no difference. You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler robot behavior -- they are the same for any robot you care about.


8

Mozilla/2.0 and Mozilla/5.0 are both references to the Mozilla browser. It has become largely meaningless, with many crawlers using it, but should tell the site to treat your crawler as it would any random user browsing with a regular browser. It is however good etiquette to include an URL linking to a page about who you are and why you are crawling in the ...


8

No, the robots.txt syntax only supports wildcards, and not regular expressions. See here for a good discussion of the syntax: http://www.robotstxt.org/robotstxt.html.


8

It would seem that Google has probably not yet updated it's cache of your robots.txt file. Your current robots.txt file (above) does not look as if it should be blocking your sitemap URL. I guess google just hasnt updated its cache. There is no need to guess. In Google Webmaster Tools (GWT) under "Health" > "Blocked URLs", you can see when your robots....


8

Great question, and one many webmasters might be interested in since the Baidu spider is notoriously aggressive and can zap resources from servers... As indicated in Baidu's Web Search news, the Baidu spider does not support the Crawl-delay notification setting, and instead requires you to register and verify your site with its Baidu Webmaster Tools ...



Only top voted, non community-wiki answers of a minimum length are eligible