Tagged Questions
2
votes
1answer
44 views
How to make “crawl-delay” works for all “user-agent” in robots.txt or in .htaccess?
I have this line:
Crawl-delay: 50
in the robots.txt file at the root of my website.
But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes.
I ...
0
votes
0answers
264 views
blocking bad bots with robots.txt in 2012 [closed]
does it still work good? I have this:
# Generated using http://solidshellsecurity.com services
# Begin block Bad-Robots from robots.txt
User-agent: asterias
Disallow:/
User-agent: BackDoorBot/1.0
...
1
vote
1answer
224 views
Need to sanity-check my .htaccess, especially Limit GET POST line for Google repellent
I need a sanity check on this .htaccess (from a WordPress site) I inherited from a 5 month+ old site.
What's the symptom? Google + Bing crawl, but don't index any of the pages.
Let me be clear: I'm ...
1
vote
1answer
297 views
Block access to robots.txt file .htaccess
I read in this forum that this .htaccess code should block access to the robot.txt file:
RewriteCond %{HTTP_USER_AGENT} ^Mozilla
RewriteCond %{HTTP_USER_AGENT} !(Slurp|surfsafely)
RewriteRule ...
2
votes
9answers
4k views
Need to Stop Bots from Killing my Webserver
I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
0
votes
1answer
147 views
Stop searches showing tld domain sites appearing as subdirectory of main holding site domain?
I'm not sure if this is a problem specific to how my hosting is structured or not.
Let's say the main domain for my account is maindomain.com (for purpose of discussion)
I am building other sites, ...
2
votes
2answers
364 views
Image Hot-links & Google's Instant Previews
I'm trying to adjust my site:
1. stop Image Hot-links
2. block Googlebot-Image (because I do not want my images to be searchable)
3. allow Google's Instant Previews
I've tried this but the Instant ...
