1

SEO is a bit of arcane topic and sometimes I encounter strange practics which seemingly aren't backed up by any advice but still persists. Now I mean something like this:

User-agent: Googlebot
Disallow: /page/
Disallow: /ajax
Disallow: *?back*
User-agent: Yandex
Disallow: /page/
Disallow: /ajax
Disallow: *?back*
...
User-agent: *
Disallow: /page/
Disallow: /ajax
Disallow: *?back*

Does this makes some sense? Those rules are really totally same.

  • Is there ever a use case where you'd want a specific bot to visit, but not others? I can think of one case if a bot needs to visit for services you signed up with (AdSense, AdWords)... – Sun Dec 18 '14 at 16:42
5

It is redundant and unnecessary.

/* Applies to all robots */
User-agent: *
Disallow: /page/
Disallow: /ajax
Disallow: *?back*
  • If /page/, /ajax, and *?back* should also be disallowed for Googlebot, you would have to repeat these rules for the User-agent: Googlebot record. – unor Dec 18 '14 at 15:46
  • The wild card *, which means all user-agents, also applies to Googlebot, no need for another separate entry. – Max Dec 19 '14 at 6:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.