Hot answers tagged robots
6
You can block bots but it depends on what you want for your web site.
You can block search engine bots if you don't want to see your web site indexed in particular search engine.
Example: Yandex is russian search engine. You can block it if your business is not targeting Russia.
You can block SEO bots if you don't want to use web analytics of it.
Example: ...
5
I don't think people should ban bing bot.
Bing has an equivalent Bing Webmaster tools at http://www.bing.com/toolbox/webmaster/
where they also have 'Crawl Settings' where you can adjust the crawl rate as seen in this video:
http://www.bing.com/videos/watch/video/bing-webmaster-tools-crawl-rate-settings/1ii1ej9jz
Googlebot is just as notorious in excessive ...
5
It would seem that Google has probably not yet updated it's cache of your robots.txt file. Your current robots.txt file (above) does not look as if it should be blocking your sitemap URL.
I guess google just hasnt updated its cache.
There is no need to guess. In Google Webmaster Tools (GWT) under "Health" > "Blocked URLs", you can see when your ...
4
Although I totally agree with danlefree and his previous answer, there are some ways to make it more difficult for bots, at least for "not-so-clever" bots.
1 - Other than IP and User-Agent strings, you could try to set a cookie and display ads only if cookie is set. Also, if the bot uses cookies, try to see if it's clever enough to clear them when it ...
4
you might be better off using your firewall instead of relying on a php script, since after all the php script will only affect php page loads and not static files like images. Plus, having php log all connections and check for >800 from the same ip for every request is going to add some serious overhead to your server, maybe even more so than the requests ...
4
While the German Wikipedia claims that all is a valid value, I've no idea where that information comes from, and German Wikipedia does not cite any source. It is AFAIK wrong, and IMHO a good example for why one should not rely on Wikipedia for this type of information.
If you want to use the robot meta tag to direct googlebot, the best information about ...
4
Your robots.txt file does not appear to conform to the robots.txt specification (you must specify the user agent before the directives and I do not believe that Noindex is a valid directive).
Consider the following updates:
User-agent: googlebot
Disallow: /Office/LocationDetails.aspx
Disallow: /office/default.aspx
Disallow: /Electronics/Communitylist.aspx
...
3
While attempting to block bots can help in free up resources and clean up your logs it’s important to note that Robots.txt and even using the meta tag on pages noindex does not actually stop bots visiting your site. They can still crawl your site occasionally to see if the denied from robots has been removed. A lot of bots don’t even use a user agent and ...
3
The behaviour for two conflicting meta tags isn't defined, as far as I know, but most likely the most restrictive rule will win.
This happens the similar case of robots.txt file vs meta robots tag. If robots.txt prevents indexing a page and meta-robots doesn't, the page will not be indexed. And if robots.txt allows a page but meta-robots blocks it, it will ...
3
First of all you can add a proper robots.txt entry to tell willing bots to ignore and not index your specific directory.
Bots should follow your robots.txt intentions, but some people (or bots) will read them on purpose to get to know what you want to hide.
So you will have to make sure your webserver does not automatically create an index of all files in ...
3
msnbot is quite prolific when it comes to spidering servers and if you have a lot of pages to index it can quite easily cripple your server. As traffic from MSN is considerably less than what Google can give it's quite common just to deny the msnbot via .htaccess, iptables or robots.txt. With Googlebot you can limit the speed quite easily in ...
3
If the majority of your content is behind a login then bots won't have access to it which will limit the amount of time they spend on your site.
Good bots, like those from major search engines, will automatically set their crawl rate to one appropriate for the site. Although there are some stories of bots hammering sites they tend to be older and you don't ...
2
IFrames are sometimes used to display content on web pages. Content
displayed via iFrames may not be indexed and available to appear in
Google's search results. We recommend that you avoid the use of
iFrames to display content. If you do include iFrames, make sure to
provide additional text-based links to the content they display, so
that ...
2
Have you looked at the Bing Webmaster Tools at all?
You can sign up for them and the process of claiming your domains is the same as for Google Webmasters Tools.
These can then give you a full list of crawl stats including links leading to 404s.
Note that if you've removed content you'll see 0 links becuase the not is requesting pages it knew about ...
2
Ask.com died as a search engine a long time ago. However, you can still submit a website's sitemap to it, if you wish. However, it will have to be done manually. If you'd like to submit your XML sitemap to Ask.com, simply type in the following in your browser's search bar: http://submissions.ask.com/ping?sitemap=http%3A//www.MYSITE.com/sitemap.xml, where ...
2
How can I block a bot like this?
There is no foolproof way for you to block illegitimate clicks on ads served with your content - if a human can click the ads, so can a bot.
Detection and mitigation is Google's responsibility, so (beyond completing your due diligence by reporting activity which may reflect poorly upon the status of your AdSense account) ...
2
It's highly unlikely you're going to find some completely universal list of UserAgents, in part because they can just be made up. Before even getting to that, though, it'd be a ridiculous amount of work. You just need to compile a few resources and then do some further searching for anything else you don't recognize. (Surprisingly, I can't find a Wikipedia ...
2
Use a robots.txt crawl delay as described in http://help.yandex.com/search/?id=1112639
Example:
User-agent: Yandex
Crawl-delay: 2 # specifies a 2 second timeout
Before you start banning this bot, you should first verify that your logs are actually Yandex and not someone else who is spoofing the user agent to look like they are yandex. A tactic used by ...
1
It is more or less the same as writing:
<meta name="robots" content="index, follow">
Interestingly, the "all" variant is not actually suggested by Google as an alternative, but it works* nonetheless.
Either command (when placed in the <head> section of your HTML code) tells search engines to index the page the tag is on, as well as crawl ...
1
It is possible though unlikely that page content would be cached for this long on some Google servers.
If you want to make sure this content goes away from Google entirely, you should sign up for a Webmasters account and verify your site. From there you can ask Google to remove the content from the cache. Check this article for more information: ...
1
Love it, Dangers...
Anyway, if you depend on overseas sales from international customers, you eliminate their ability to find your site through their preferred search. Eliminating Yandex, for example will mean that Russian Federation and Eastern European countries will be finding results elsewhere as you are now invisible to any searches through that SE.
...
1
Yes, crawlers will recognise, crawl and index your rewritten links as long as they are included throughout your site. You should ensure that only the re-written links are included in the source and not the non-rewritten links.
You could improve the optimisation of these links further by including the article title as part of the URL.
No problem with ...
1
Thanks for the great code. I have modified it in such a way that it will ban anything without a session for 1 whole hour if this trap is activated. I am calling this the Extreme Flood Trap. I know I will be banning even the good bots which leads me to my next question. Will serving the Searchbots with 403 for an hour have bad consequences for the site in the ...
1
I'm not sure I 100% understand your question (you should add a snippet of your robots.txt rules to get a definitive answer)
Sill, from what I read here you must have a "Disallow:\" rule inside which you'll need to remove. I`m saying this because you robot.txt is always should be in root directory and this is the only rule that can prevent access to that ...
1
This week our company (Incapsula) launched Botopedia.org - a Community-Sourced bot directory. It's 100% free and open for all and you can use it to find a complete user-agent list for all bots you`ll want to look up.
As for indentification methods, I want to refer you to this discussion in Security.Stackexchange which covers different methods of bot ...
1
One of my clients was doing $10,000 monthly from Bing shopping alone. Organics from Bing was even more. Banning them would cause a big loss of revenue. Anyone suggesting it must have their own personal reasons. Bing generates visits so if you want to decrease your traffic go a head and ban Bing. Otherwise like Anthony said you can work with their Webmaster ...
1
robots.txt only gives instructions, it doesn't enforces any kind of security.
It exist in order to tell search engines what should/shouldn't be listed, nothing more.
Lets say you have and admin.php script located under sub-folder1. A bad bot could "guess" and try to load it. If isn't password protected, you are in problems.
1
It depends. Pull up that URL in your browser. Do you see those folders listed? If so, they can find them.
To prevent these folders from being listed just add an htaccess file in that directory with this line in it:
Options -Indexes
This will prevent the listing of that directory's contents from appearing.
1
If you are limiting access to a minority of users then a whitelist is the safest route to take. It's easy to manage and literally delimits who is allowed access. For example, I use a MAC Address whitelist for my wireless router since very few people actually need access to it and they are all known quantities.
If you are dealing with a large audience, like ...
1
This may seem a little ridiculous, but if you're getting the traffic you say you are from the SAME IP addresses, would it be possible to install Fail2Ban on the server itself? (not Apache module). Some info: http://www.fail2ban.org/wiki/index.php/Main_Page but this scans Error logs from the server (Apache's logs, for example) and finds patterns where someone ...
Only top voted, non community-wiki answers of a minimum length are eligible
