For my website I tried to make a robot.txt file so my SEO would be better. After adding the robot.txt to my website I expected it to work like it should, but when I checked my SEO with an online site, it showed me it doesn't work.

My robot.txt file:

User-agent: Google

User-agent: *
Disallow: /

Can you tell me what's wrong with this?

  • 1
    When you say something "doesn't work", you need to explain (in natural language) what it is you are trying to do, otherwise we can do little more the check for basic syntax errors in your code. It still may not be doing what you intend it to do. – DocRoot Jun 15 '18 at 10:09

The problem

You have called the file robot.txt, it should be called robots.txt.

Other problems

  • You have used the tag Google, but GoogleBot uses the tag Googlebot [Source].
  • 1
    Disallow: and Allow: / are the same thing, you can use either for Googlebot. However, not all bots understand allow directives (google does), so its generally best to keep to robots.txt protocol if its for all bots, which doesn't include allow. – Max Jun 15 '18 at 9:48
  • "You have used the tag Google, but GoogleBot uses the tag GoogleBot" - Rather curious, I would have expected the shorter identifier "Google" to still catch the Googlebot (it is after all just a shorter pattern). However, testing in Google's robots.txt tester "Google" fails to match the Googlebot UA!?. (Strictly speaking, it's "Googlebot", not "GoogleBot" - but the match is non-case-sensitive anyway.) – DocRoot Jun 15 '18 at 10:59
  • Google​ has a few different bots and user agents, so it needs to be exact so you can block just what you want – Max Jun 15 '18 at 11:30
  • 1
    The original spec recommends (but doesn’t require) that bots also recognize substring matches in User-agent values. – unor Jun 15 '18 at 12:11
  • 1
    @unor Although "Googlebot" is already arguably a substring. It seems that, in the case of Google at least, specific user-agent "tokens" are matched exactly. I've added an answer to this related StackOverflow question, referenced from your link, with some findings. The Google Crawlers page is also a good reference as it lists these "tokens". – DocRoot Jun 15 '18 at 17:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.