Hot answers tagged web-crawlers
6
This is no doubt dependent on your remaining 200+ lines, but what you have posted so far would seem to be reducible to just 2 lines:
RewriteRule ^patients/billing/(FAQ_billing|getintouch).html$ $1.php [L,NC]
RewriteRule ^patients/findadoctor/([a-z]).html$ findadoctor.php?id=$1 [L,NC]
This shouldn't be reducing your site to a crawl.
As mentioned in ...
2
I know you previously asked to block the bots and now you just want to slow them down.
The only way to force them to slow down is not to serve them content if they are going too fast. The problem with not serving content is it will surely have negative consequences to your search engine rankings.
In case you decide to block some of the less important bots, ...
1
The htaccess does, by definition, slow down your site because the server must check every folder for the file - e.g. in your case the folders /patients/findadoctor/, /patients/ and the root folder. It is more efficient to write those rules in the server config itself, then set AllowOverride None, which will stop Apache looking for htaccess files. If you are ...
Only top voted, non community-wiki answers of a minimum length are eligible
