Tag Info

New answers tagged

21

The last record (started by User-agent: *) will be followed by all polite bots that don’t identify themselves as "googlebot", "google", "bingbot" or "bing". And yes, it means that they are not allowed to crawl anything. You might want to omit the * in /bedven/bedrijf/*. In the original robots.txt specification, * has no special meaning, it’s just a ...


2

If you literally only have a few "groups" you want to block then you would do something like: User-agent: * Disallow: /lang/group1 Disallow: /lang/group2 ...and everything else would be allowed. This would work with all robots that obey the original "standard". Or, you could block all groups (group1, group2, etc.) and make an exception for "group3", like: ...


1

I think the problem with your rewrite rule is the OR flag. That flag usually means that there is a second rewrite condition coming. You only have one condition. Here is a site that provides a similar rule for blocking BaiduSpider with slightly different syntax: RewriteEngine On RewriteCond %{HTTP_USER_AGENT} ^Baiduspider.* [NC] RewriteRule .* - [F]


1

You can try blocking specific IP addresses in your .htaccess file. You can find the ranges here. In robots.txt you can also add the following User-agent: Baiduspider User-agent: baiduspider User-agent: Baiduspider+ User-agent: Baiduspider-video User-agent: Baiduspider-image Disallow: / Also, if you use caching plugins or CDN, make sure to clear all your ...



Top 50 recent answers are included