Tag Info

Hot answers tagged

22

Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt. Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.


16

Apparently it does, some, according to Matt Cutts: "For a while, we were scanning within JavaScript, and we were looking for links. Google has gotten smarter about JavaScript and can execute some JavaScript. I wouldn't say that we execute all JavaScript, so there are some conditions in which we don't execute JavaScript. Certainly there are some common, ...


14

I've used Xenu's Link Sleuth. It works pretty well, just be sure not to DOS yourself!


14

I'm the primary designer and author of a fairly large-scale web crawler (see http://metadatalabs.com/mlbot). What you're asking touches on a topic that's very important to us--perhaps the most important part of running a crawler: that of politeness. First: the reason for the "Mozilla" thing is to tell the site what your browser capabilities are. If your ...


13

This is Vincent from Bing Webmaster Tools. This was a Bingbot checking for an XML sitemap generated by the Bing Sitemap Plugin (Beta) for IIS and Apache - see the following for more details: Bing Sitemap Plugin (Beta) - Bing Webmaster Tools Help & How-To Center Building Sitemaps Manually?… Stop Until You Read This! - Bing Webmaster Blog You ...


13

Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure If your site is password protected properly, there shouldn't be any need to block particular user agents :) x


13

Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic. Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...


11

Yes and no, he's conflating two different things. In late 90's/early 2000's, when designers needed to use fonts for design reasons, they embedded graphics files in HTML pages. Search engines cannot understand these, nor can the screen readers used by blind and partially sighted people. This is because the only HTML code in the page for a menu link, for ...


11

We now utilize a crawling method of adding new content to our search index. The subdomain you refer to was just retired over the weekend, and we'll be updating that FAQ page very shortly. Best regards, Eric from Ask.com


11

Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless ...


10

I would see no reason to unblock Yahoo's search bot. There are 2 ways Yahoo could be using the Bing Engine. They could integrate it with their own and set a hybrid crawler loose on the net or they could just be sending the queries into the Bing Engine and spitting out the results they get back. From the document Joel linked to there is another document ...


10

You do not need one as not having one is interpreted as meaning you want to have all of your content crawled. But I recommend using a blank one just to prevent the accumulation of unnecessary 404 errors in your stats.


10

It's not ARPA (Advanced Research Project Agency). That's just DNS at work. Here is a write-up on The in-addr.arpa Domain.


9

I'm glad you know there isn't going to be a bulletproof way to accomplish this. That means your outlook is at least realistic. Since JavaScript is not an option I would say you're left with: Check the user-agent for the word "bot" in it. That will catch most of them. Compile a list of known bots and filter them based on some kind of unique identifier, ...


9

You can add a dollar sign to the end of the string which means it will only match exactly that entry: # Files User-agent: * Disallow: /mage$ This will only block the mage file if it come straight after the root domain: www.example.com/mage If there are any other preceding directories, you must add these o the entry. So to block the file located ...


8

No, the robots.txt syntax only supports wildcards, and not regular expressions. See here for a good discussion of the syntax: http://www.robotstxt.org/robotstxt.html.


8

Can they see it? Yes. Can they find it? Not without help. Web crawlers typically find pages to crawl by following links to them on other pages. Some crawlers (e.g. search engine crawlers) will also crawl pages listed in special XML files. So if there is no link to page on your website or any other website then that page will not be crawled (pages that ...


8

It would seem that Google has probably not yet updated it's cache of your robots.txt file. Your current robots.txt file (above) does not look as if it should be blocking your sitemap URL. I guess google just hasnt updated its cache. There is no need to guess. In Google Webmaster Tools (GWT) under "Health" > "Blocked URLs", you can see when your ...


7

It appears that they've just started rolling out Bing search results on Yahoo! for about 25% of the searches, with the goal of completely switching over in August/September. I would bet that the Yahoo spider isn't even running any more and if it is, it's not very relevant.


7

Mozilla/2.0 and Mozilla/5.0 are both references to the Mozilla browser. It has become largely meaningless, with many crawlers using it, but should tell the site to treat your crawler as it would any random user browsing with a regular browser. It is however good etiquette to include an URL linking to a page about who you are and why you are crawling in the ...


7

Yes, if the site requires authentication then robots will not be able to crawl it. You can also specify a robots.txt file so that the entire site is disallowed


7

You can link to a hidden page that, when visited, captures the useragent and IP address of the bot and then appends one or both of them to a .htaccess file which blocks them permanently. It's automated so you don't have to do anything to maintain it.


7

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


6

If you are using windows 7 the best tool is IIS7's SEO Toolkit 1.0. It is free and you can download it for free. The tool will scan any site and tell you where all of the dead links are, what pages take to long to load, what pages have missing titles, duplicate titles, same for keywords and descriptions, and what pages have broken HTML.


6

Yes. You can disallow everything first, then allow the folders you want to be indexed. User-agent: * Disallow: / Allow: /index.html Allow: /example/ Allow: /example2/ Allow: /example3/ The reason this works is that Google (and Bing) follow CSS-style specificity rules when reading robots.txt files, where more specific rules take precedence over less ...


6

Allow is non-standard according to Wikipedia: http://en.wikipedia.org/wiki/Robots.txt.


6

If it's a HTML document, include the following meta tag in it: <meta name="robots" content="noindex" /> This will tell Google and every other search engine that understands the tag (which should be pretty much all of them, since it's been around since 1996) not to index the page. For non-HTML documents, you can configure your webserver to send the ...


6

I don't think people should ban bing bot. Bing has an equivalent Bing Webmaster tools at http://www.bing.com/toolbox/webmaster/ where they also have 'Crawl Settings' where you can adjust the crawl rate as seen in this video: http://www.bing.com/videos/watch/video/bing-webmaster-tools-crawl-rate-settings/1ii1ej9jz Googlebot is just as notorious in excessive ...


6

You can block bots but it depends on what you want for your website. You can block search engine bots if you don't want to see your website indexed in a particular search engine. Example: Yandex is russian search engine. You can block its bot if your business is not targeting Russia. You can block SEO bots if you don't want to use their web analytics ...


6

Google has no knowledge of the actual file structure of your web app and what is accessible to the public. A news site, for example, will have hundreds of articles to view, but might only have one actual script: article.php. Server-side directives might allow pretty canonical URLs like /category/283423-pretty-name to point to /article.php, and this is what ...



Only top voted, non community-wiki answers of a minimum length are eligible