Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.
Sign up to join this community
I have a small website, for which the current robots.txt looks like so:
User-agent: *
Disallow:
Sitemap: https://www.myawesomesite.com/sitemap.xml
Now, I want to add another rule like so (with the aim to disallow SomeStupidBot for crawling my website):
User-agent: SomeStupidBot
Disallow: /
Sitemap: https://www.myawesomesite.com/sitemap.xml
I am unsure how I combine the above two rules into one robots.txt.
To combine them you simply append the second rule block (group) to the file, with a line gap, like so:
User-agent: *
Disallow:
User-agent: SomeStupidBot
Disallow: /
Sitemap: https://www.myawesomesite.com/sitemap.xml
The order does not matter (the most specific - based on length - wins). Only one block is processed by the bot.
Note that the Sitemap directive is not part of the group. It is an independent directive that often appears at the end of the file (although the location does not really matter).
As @MEGA has already stated, the first group is superfluous in this instance. However, it is up to you whether you keep it (there will usually be something you don't want other bots to crawl).
By default, all bots are allowed to crawl all parts of your site with or without the first code you provided.
So to disallow one bot just add the second code:
User-agent: SomeStupidBot
Disallow: /
and remove:
User-agent: *
Disallow:
I am not an expert in robots.txt, but from my understanding this is how its done.
