1

I'm having problem with robots.txt. I put the robots.txt file in the website main directory (and also in /var/www/html - to make it work on all the server) but robots still keep crawling my websites.

this is my robots.txt:

User-agent: YandexBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot/1.2~bl
Disallow: /

Do you have any suggestions?

  • Which of those robots are still crawling your site? Not all bots obey robots.txt, but I'd be surprised if any of those particular ones were problematic. – Stephen Ostermiller Dec 6 '17 at 10:35
  • Have you checked that your robots.txt file is actually being served for your site? Can you visit http://example.com/robots.txt and get your file? (Of course substituting your domain name where I have example.com.) – Stephen Ostermiller Dec 6 '17 at 10:36
  • How long have you waited after putting the robots.txt file in place? Crawlers need to re-fetch the robots.txt to realize they shouldn't be crawling. No crawler fetches robots.txt for every page they crawl. Most fetch it once a day. – Stephen Ostermiller Dec 6 '17 at 10:37
  • 1
    Hello,i've checked and the robots.txt is public to any who wants to view the file. ive made this robots.txt file one day ago. – chen Dec 6 '17 at 12:02
2

Note that your robots.txt is invalid (but that doesn’t necessarily mean that this is the reason for the issue you are having; bots might ignore such errors).

If a bot parses your robots.txt file strictly according to the robots.txt spec, this bot would only see one record, and this record would only apply to bots with the name "YandexBot". All other bots would be allowed to crawl everything.

The reason is that records must be separated with blank lines. So it should be:

User-agent: YandexBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot/1.2~bl
Disallow: /

If you’ll always have the same Disallow for all these bots, you could use one record with multiple User-agent lines, if you prefer it:

User-agent: YandexBot
User-agent: SemrushBot
User-agent: AhrefsBot
User-agent: SemrushBot/1.2~bl
Disallow: /

(You might have to use different names for some of the bots you intend to block, as @StephenOstermiller suggests in his answer.)

1

After you create your robots.txt file it will take a day or more for the crawlers that honor it to fetch it.

Yandex has a number of bots and have documentation about how to disallow all of them using robots.txt here: https://yandex.com/support/webmaster/controlling-robot/robots-txt.xml You might want to consider changing your robots.txt to this for Yandex:

User-agent: Yandex
Disallow: /

SEM Rush has two bots. Their documentation about it is here: https://www.semrush.com/bot/ You have disallowed one of them correctly but your second rule with the version number of the bot will not be effective. You should consider using these rules to disallow all SEM Rush crawling:

User-agent: SemrushBot
Disallow: / 

User-agent: SemrushBot-SA
Disallow: / 

You are already disallowing AhrefsBot exactly according to their documentation: https://ahrefs.com/robot

User-agent: AhrefsBot
Disallow: / 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.