Hot answers tagged web-crawlers
Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt. Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.
The last record (started by User-agent: *) will be followed by all polite bots that don’t identify themselves as "googlebot", "google", "bingbot" or "bing". And yes, it means that they are not allowed to crawl anything. You might want to omit the * in /bedven/bedrijf/*. In the original robots.txt specification, * has no special meaning, it’s just a ...
Do crawlers behave differently in these two cases? A robots.txt file that's empty is really no different from one that's not found, both do not disallow crawling. You might however receive lots of 404 errors in your server logs when crawlers request the robots.txt file, as indicated in this question here. So, is it safe to just delete an empty ...
I'm the primary designer and author of a fairly large-scale web crawler (see http://metadatalabs.com/mlbot). What you're asking touches on a topic that's very important to us--perhaps the most important part of running a crawler: that of politeness. First: the reason for the "Mozilla" thing is to tell the site what your browser capabilities are. If your ...
I've used Xenu's Link Sleuth. It works pretty well, just be sure not to DOS yourself!
Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure If your site is password protected properly, there shouldn't be any need to block particular user agents :) x
Banning bots will not let any search engine get the content of the site. Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic. Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like ...
This is Vincent from Bing Webmaster Tools. This was a Bingbot checking for an XML sitemap generated by the Bing Sitemap Plugin (Beta) for IIS and Apache - see the following for more details: Bing Sitemap Plugin (Beta) - Bing Webmaster Tools Help & How-To Center Building Sitemaps Manually?… Stop Until You Read This! - Bing Webmaster Blog You ...
Yes and no, he's conflating two different things. In late 90's/early 2000's, when designers needed to use fonts for design reasons, they embedded graphics files in HTML pages. Search engines cannot understand these, nor can the screen readers used by blind and partially sighted people. This is because the only HTML code in the page for a menu link, for ...
We now utilize a crawling method of adding new content to our search index. The subdomain you refer to was just retired over the weekend, and we'll be updating that FAQ page very shortly. Best regards, Eric from Ask.com
Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless ...
I would see no reason to unblock Yahoo's search bot. There are 2 ways Yahoo could be using the Bing Engine. They could integrate it with their own and set a hybrid crawler loose on the net or they could just be sending the queries into the Bing Engine and spitting out the results they get back. From the document Joel linked to there is another document ...
You do not need one as not having one is interpreted as meaning you want to have all of your content crawled. But I recommend using a blank one just to prevent the accumulation of unnecessary 404 errors in your stats.
It's not ARPA (Advanced Research Project Agency). That's just DNS at work. Here is a write-up on The in-addr.arpa Domain.
There are at least 3 ways: Links to your site. Using Google Webmaster Tools (now called Search Console) Registrar dumps, triggers, and other options. Google will find many new sites quickly from some registrars. For example, Google found one domain name I registered using GoDaddy, indexed it, and began sending search results within 20 minutes of ...
You can add a dollar sign to the end of the string which means it will only match exactly that entry: # Files User-agent: * Disallow: /mage$ This will only block the mage file if it come straight after the root domain: www.example.com/mage If there are any other preceding directories, you must add these o the entry. So to block the file located ...
Mozilla/2.0 and Mozilla/5.0 are both references to the Mozilla browser. It has become largely meaningless, with many crawlers using it, but should tell the site to treat your crawler as it would any random user browsing with a regular browser. It is however good etiquette to include an URL linking to a page about who you are and why you are crawling in the ...
No, the robots.txt syntax only supports wildcards, and not regular expressions. See here for a good discussion of the syntax: http://www.robotstxt.org/robotstxt.html.
Can they see it? Yes. Can they find it? Not without help. Web crawlers typically find pages to crawl by following links to them on other pages. Some crawlers (e.g. search engine crawlers) will also crawl pages listed in special XML files. So if there is no link to page on your website or any other website then that page will not be crawled (pages that ...
It would seem that Google has probably not yet updated it's cache of your robots.txt file. Your current robots.txt file (above) does not look as if it should be blocking your sitemap URL. I guess google just hasnt updated its cache. There is no need to guess. In Google Webmaster Tools (GWT) under "Health" > "Blocked URLs", you can see when your ...
No. There's no difference. You'd get 404 errors in your server log, and if you're subscribed to things like Google Web Master tools it might tell you you've not got one, but in terms of the crawler robot behavior -- they are the same for any robot you care about.
It appears that they've just started rolling out Bing search results on Yahoo! for about 25% of the searches, with the goal of completely switching over in August/September. I would bet that the Yahoo spider isn't even running any more and if it is, it's not very relevant.
Yes, if the site requires authentication then robots will not be able to crawl it. You can also specify a robots.txt file so that the entire site is disallowed
You can link to a hidden page that, when visited, captures the useragent and IP address of the bot and then appends one or both of them to a .htaccess file which blocks them permanently. It's automated so you don't have to do anything to maintain it.
Allow is non-standard according to Wikipedia: http://en.wikipedia.org/wiki/Robots.txt.
If it's a HTML document, include the following meta tag in it: <meta name="robots" content="noindex" /> This will tell Google and every other search engine that understands the tag (which should be pretty much all of them, since it's been around since 1996) not to index the page. For non-HTML documents, you can configure your webserver to send the ...
Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
Google expects differences between mobile and desktop sites. Even major differences, including differences in link structure, are not a problem. Google crawls the web with different Googlebot user agents for mobile. As long as your server shows that version of Googlebot the same thing that your actual mobile users see, you don't have any penalty risk. ...
Only top voted, non community-wiki answers of a minimum length are eligible