Tagged Questions
2
votes
2answers
26 views
Question about duplicate groups (agents) in robots.txt
I'm looking at a corner case with robots.txt, and am curious if you have any guidance.
Basically, I'm in a situation where multiple people will be making edits to the same robots.txt file.
I've ...
0
votes
1answer
247 views
How to secure robots.txt file?
I would like for User-agents to index my relative pages only without accessing any directory on my server.
As initial thought, i had this version in mind:
User-agent: *
Disallow: */*
Sitemap: ...
3
votes
1answer
531 views
Preventing high bandwidth usage from Yandex
Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
1
vote
1answer
297 views
Block access to robots.txt file .htaccess
I read in this forum that this .htaccess code should block access to the robot.txt file:
RewriteCond %{HTTP_USER_AGENT} ^Mozilla
RewriteCond %{HTTP_USER_AGENT} !(Slurp|surfsafely)
RewriteRule ...
