1

I have a small website, for which the current robots.txt looks like so:

User-agent: *
Disallow:
Sitemap: https://www.myawesomesite.com/sitemap.xml

Now, I want to add another rule like so (with the aim to disallow SomeStupidBot for crawling my website):

User-agent: SomeStupidBot
Disallow: /
Sitemap: https://www.myawesomesite.com/sitemap.xml

I am unsure how I combine the above two rules into one robots.txt.

  • If a bot is really stupid then you can't block it via robots.txt, you have to check your site logs and based on IP address + user-agent you need to block it on htaccess. But most of bots honor your robots.txt and they might also have docs pages to know how to block it. – Goyllo Sep 9 '17 at 7:41
2

To combine them you simply append the second rule block (group) to the file, with a line gap, like so:

User-agent: *
Disallow:

User-agent: SomeStupidBot
Disallow: /

Sitemap: https://www.myawesomesite.com/sitemap.xml

The order does not matter (the most specific - based on length - wins). Only one block is processed by the bot.

Note that the Sitemap directive is not part of the group. It is an independent directive that often appears at the end of the file (although the location does not really matter).

As @MEGA has already stated, the first group is superfluous in this instance. However, it is up to you whether you keep it (there will usually be something you don't want other bots to crawl).

1

By default, all bots are allowed to crawl all parts of your site with or without the first code you provided.
So to disallow one bot just add the second code:

User-agent: SomeStupidBot
Disallow: /

and remove:

User-agent: *
Disallow:

I am not an expert in robots.txt, but from my understanding this is how its done.

  • 1
    This would work, but I'd recommend keeping both sets of rules. That way you can add disallow rules for all bots easily. – Stephen Ostermiller Sep 8 '17 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.