1

I am about to launch a new WordPress site and below is what I came up with for my robots.txt file. Should I combine some of these directives with a wildcard or should I include them both ways? I know that some search engines don't read wildcards.

sitemap: http://www.example.com/sitemap.xml

User-agent:  *
Disallow: */comments/
Disallow: */feed/
Disallow: */trackback/
Disallow: *?replytocom
Disallow: /*.css$
Disallow: /*.inc$
Disallow: /*.js$
Disallow: /*.php$
Disallow: /*/*.inc$
Disallow: /*/*.js$
Disallow: /*/*?s=*
Disallow: /*/cgi-bin/*
Disallow: /*/feed/
Disallow: /*/feed/*
Disallow: /*/trackback/
Disallow: /*/trackback/*
Disallow: /*/wp-*
Disallow: /*/xmlrpc.php
Disallow: /*?
disallow: /*?*
Disallow: /archives/
Disallow: /author
Disallow: /cgi-bin/
Disallow: /comments/
Disallow: /comments/feed/
Disallow: /feed/
Disallow: /index.php
Disallow: /login/
Disallow: /wp-*
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-content/cache/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-includes/
Disallow: /wp-rss.php
Disallow: /wp-rss2.php
Disallow: /xmlrpc.php
Disallow: ?wptheme=
Disallow: /readme.html
Disallow: /license.txt
Allow: /wp-content/uploads

User-agent: Mediapartners-Google*
Disallow:
Allow: /

User-agent: Googlebot-Image
Disallow:
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Disallow:
Allow: /

User-agent: Googlebot-Mobile
Disallow:
Allow: /

User-agent: duggmirror
Disallow: /

User-agent: ia_archiver
Disallow: /
1
  • 2
    Unfortunately, users won't know what you're trying to accomplish here, so only general opinions could be offered. If you can edit this to a specific concern that you have (e.g., "Is the following correct to do ___",) then it will be reviewed to be reopened. – dan May 23 '14 at 3:07
1

I think this one Disallow: /wp-* should be included:

Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-content/cache/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-includes/
Disallow: /wp-rss.php
Disallow: /wp-rss2.php

So you need Disallow: /wp-* only, not all.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.