Hot answers tagged robots
6
You can block bots but it depends on what you want for your web site.
You can block search engine bots if you don't want to see your web site indexed in particular search engine.
Example: Yandex is russian search engine. You can block it if your business is not targeting Russia.
You can block SEO bots if you don't want to use web analytics of it.
Example: ...
5
I'm glad you know there isn't going to be a bulletproof way to accomplish this. That means your outlook is at least realistic.
Since JavaScript is not an option I would say you're left with:
Check the user-agent for the word "bot" in it. That will catch most of them.
Compile a list of known bots and filter them based on some kind of unique identifier, ...
5
I don't think people should ban bing bot.
Bing has an equivalent Bing Webmaster tools at http://www.bing.com/toolbox/webmaster/
where they also have 'Crawl Settings' where you can adjust the crawl rate as seen in this video:
http://www.bing.com/videos/watch/video/bing-webmaster-tools-crawl-rate-settings/1ii1ej9jz
Googlebot is just as notorious in excessive ...
5
It would seem that Google has probably not yet updated it's cache of your robots.txt file. Your current robots.txt file (above) does not look as if it should be blocking your sitemap URL.
I guess google just hasnt updated its cache.
There is no need to guess. In Google Webmaster Tools (GWT) under "Health" > "Blocked URLs", you can see when your ...
5
you might be better off using your firewall instead of relying on a php script, since after all the php script will only affect php page loads and not static files like images. Plus, having php log all connections and check for >800 from the same ip for every request is going to add some serious overhead to your server, maybe even more so than the requests ...
4
I'm the primary designer and author of a fairly large-scale web crawler (see http://metadatalabs.com/mlbot). What you're asking touches on a topic that's very important to us--perhaps the most important part of running a crawler: that of politeness.
First: the reason for the "Mozilla" thing is to tell the site what your browser capabilities are. If your ...
4
Mozilla/2.0 and Mozilla/5.0 are both references to the Mozilla browser. It has become largely meaningless, with many crawlers using it, but should tell the site to treat your crawler as it would any random user browsing with a regular browser.
It is however good etiquette to include an URL linking to a page about who you are and why you are crawling in the ...
4
There are three fairly simple ways:
Use Google Analytics, which will process and handle all the data for you, and present you with detailed statistics for visitors and how they got to your site. This is by far the easiest solution.
Use Javascript to do the counting. When the page has loaded, generate an AJAX request to your counting script. Robots and ...
4
First, the "Allow" directive in your robots.txt does nothing as robots spider everything by default.
Blocking robots from *.* is probably OK in some situations, but remember that you are blocking every URL that simply contains a dot. A more reliable method may be blocking individual extensions (if there are not too many) eg *.html and *.php on separate ...
4
It's depends on which search engine you refer to. Most people are interested in google so I'll assume that's what you meant:
Generally speaking, the higher your rank is, and if the frequency in which you change your site (add articles etc) is high - your site will be crawled more frequently.
The following article might interest you.
4
Although I totally agree with danlefree and his previous answer, there are some ways to make it more difficult for bots, at least for "not-so-clever" bots.
1 - Other than IP and User-Agent strings, you could try to set a cookie and display ads only if cookie is set. Also, if the bot uses cookies, try to see if it's clever enough to clear them when it ...
4
Your robots.txt file does not appear to conform to the robots.txt specification (you must specify the user agent before the directives and I do not believe that Noindex is a valid directive).
Consider the following updates:
User-agent: googlebot
Disallow: /Office/LocationDetails.aspx
Disallow: /office/default.aspx
Disallow: /Electronics/Communitylist.aspx
...
4
While the German Wikipedia claims that all is a valid value, I've no idea where that information comes from, and German Wikipedia does not cite any source. It is AFAIK wrong, and IMHO a good example for why one should not rely on Wikipedia for this type of information.
If you want to use the robot meta tag to direct googlebot, the best information about ...
3
Yes, the IP addresses in your logs probably belong to bots.
If the bot is set to only download html, and not request any other resources (javascript, images, etc), each page view will correspond to a single hit.
Most bots (indexing spiders, screen-scrapers, vulnerability hunters) will only be interested in the HTML, so this looks like typical bot ...
3
Bit of digging turns up this old thread from Webmaster World, another thread from ServerFault. The conclusion seems to be that it's caused by a bot looking for installations of a Bluecoat product called ProxySG to attack.
Seems a security product called ProxySG uses requests like
verify-SNL_Splash. Meant to be on the LAN side but apparently it may
be ...
3
Yes. You can disallow everything first, then allow the folders you want to be indexed.
User-agent: *
Disallow: /
Allow: /index.html
Allow: /example/
Allow: /example2/
Allow: /example3/
The reason this works is that Google (and Bing) follow CSS-style specificity rules when reading robots.txt files, where more specific rules take precedence over less ...
3
It's a good thing. This means that Google will find your new content faster which means it can be included in their index sooner. It's got to be in the index to be found in searches.
Keep in mind that this doesn't mean your pages will rank well. It doesn't even necessarily mean your pages will be indexed. But under normal circumstances being visited ...
3
Webmaster Tools used to show you all the URLs that you have blocked with robots.txt (under Crawl Errors), however that functionality appears to no longer exist. There is only the Crawler Access section that lists how many URLs are blocked.
If your pages are appearing in search results without problems (a quick site: search shows that is the case) then there ...
3
If the majority of your content is behind a login then bots won't have access to it which will limit the amount of time they spend on your site.
Good bots, like those from major search engines, will automatically set their crawl rate to one appropriate for the site. Although there are some stories of bots hammering sites they tend to be older and you don't ...
3
msnbot is quite prolific when it comes to spidering servers and if you have a lot of pages to index it can quite easily cripple your server. As traffic from MSN is considerably less than what Google can give it's quite common just to deny the msnbot via .htaccess, iptables or robots.txt. With Googlebot you can limit the speed quite easily in ...
3
First of all you can add a proper robots.txt entry to tell willing bots to ignore and not index your specific directory.
Bots should follow your robots.txt intentions, but some people (or bots) will read them on purpose to get to know what you want to hide.
So you will have to make sure your webserver does not automatically create an index of all files in ...
3
While attempting to block bots can help in free up resources and clean up your logs it’s important to note that Robots.txt and even using the meta tag on pages noindex does not actually stop bots visiting your site. They can still crawl your site occasionally to see if the denied from robots has been removed. A lot of bots don’t even use a user agent and ...
3
The behaviour for two conflicting meta tags isn't defined, as far as I know, but most likely the most restrictive rule will win.
This happens the similar case of robots.txt file vs meta robots tag. If robots.txt prevents indexing a page and meta-robots doesn't, the page will not be indexed. And if robots.txt allows a page but meta-robots blocks it, it will ...
2
All of the search engines use a huge number of IP addresses. You'll want to look at the user agent string instead. Check this page for a good list of all crawlers.
In PHP, something like this would work:
$bots = array( 'googlebot', 'msnbot', 'slurp', 'mediapartners-google' );
$isRobot = false;
$ua = strtolower( $_SERVER['HTTP_USER_AGENT'] );
foreach ( ...
2
If you use Javascript to count views then bots won't run it and so won't be included in your view counts. This answer may be close to what you want http://stackoverflow.com/questions/1973448/how-can-i-count-a-page-views
2
Obvious disadvantage of blacklisting is blocking valid users sharing NAT with machines you want to blacklist (think: big corporations with VPN and only few external IPs or universities with similar setups, maybe even some ISPs).
In particular case of worms, such as the one trying to propagate through bugs in phpMyAdmin, that would mean you blacklist ...
2
Robots.txt is an advisory protocol; no-one has to follow it, and several spiders don't (e.g. those used by Internet archivists like the archive team.)
As such, you might like to consider whitelisting the IP addresses that will be accessing your real phpMyAdmin area and excluding all others, rather than blacklisting all those who hit your honeypot. The ...
2
"Hammer" is subjective, but there does seem to be a general belief that Slurp is let's say...maybe a little more aggressive than it really needs to be, and that it's been going on for years, ie. 2007, 2009, 2011. A basic search for something like slurp robots delay will turn up a giant pile of complaints.
In case you're tempted to actually block Slurp, as ...
2
Let's try an empirical approach.
In the access logs for my site, I see 55 sitemap requests over the last 33 days. Out of those 55, 30 are from Googlebot, 21 from msnbot and the remaining four are from Exabot. (I've only submitted the sitemap manually to Google; the others have found it through robots.txt.)
So that's one data point for about "every day", ...
2
Have you looked at the Bing Webmaster Tools at all?
You can sign up for them and the process of claiming your domains is the same as for Google Webmasters Tools.
These can then give you a full list of crawl stats including links leading to 404s.
Note that if you've removed content you'll see 0 links becuase the not is requesting pages it knew about ...
Only top voted, non community-wiki answers of a minimum length are eligible
