1

I am about to launch a new WordPress site and below is what I came up with for my robots.txt file. Should I combine some of these directives with a wildcard or should I include them both ways? I know that some search engines don't read wildcards.

sitemap: http://www.example.com/sitemap.xml

User-agent:  *
Disallow: */comments/
Disallow: */feed/
Disallow: */trackback/
Disallow: *?replytocom
Disallow: /*.css$
Disallow: /*.inc$
Disallow: /*.js$
Disallow: /*.php$
Disallow: /*/*.inc$
Disallow: /*/*.js$
Disallow: /*/*?s=*
Disallow: /*/cgi-bin/*
Disallow: /*/feed/
Disallow: /*/feed/*
Disallow: /*/trackback/
Disallow: /*/trackback/*
Disallow: /*/wp-*
Disallow: /*/xmlrpc.php
Disallow: /*?
disallow: /*?*
Disallow: /archives/
Disallow: /author
Disallow: /cgi-bin/
Disallow: /comments/
Disallow: /comments/feed/
Disallow: /feed/
Disallow: /index.php
Disallow: /login/
Disallow: /wp-*
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-content/cache/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-includes/
Disallow: /wp-rss.php
Disallow: /wp-rss2.php
Disallow: /xmlrpc.php
Disallow: ?wptheme=
Disallow: /readme.html
Disallow: /license.txt
Allow: /wp-content/uploads

User-agent: Mediapartners-Google*
Disallow:
Allow: /

User-agent: Googlebot-Image
Disallow:
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Disallow:
Allow: /

User-agent: Googlebot-Mobile
Disallow:
Allow: /

User-agent: duggmirror
Disallow: /

User-agent: ia_archiver
Disallow: /
1
  • 2
    Unfortunately, users won't know what you're trying to accomplish here, so only general opinions could be offered. If you can edit this to a specific concern that you have (e.g., "Is the following correct to do ___",) then it will be reviewed to be reopened.
    – dan
    Commented May 23, 2014 at 3:07

1 Answer 1

1

I think this one Disallow: /wp-* should be included:

Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-content/cache/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-includes/
Disallow: /wp-rss.php
Disallow: /wp-rss2.php

So you need Disallow: /wp-* only, not all.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.