Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I am about to launch a new WordPress site and below is what I came up with for my robots.txt file. Should I combine some of these directives with a wildcard or should I include them both ways? I know that some search engines don't read wildcards.

sitemap: http://www.example.com/sitemap.xml

User-agent:  *
Disallow: */comments/
Disallow: */feed/
Disallow: */trackback/
Disallow: *?replytocom
Disallow: /*.css$
Disallow: /*.inc$
Disallow: /*.js$
Disallow: /*.php$
Disallow: /*/*.inc$
Disallow: /*/*.js$
Disallow: /*/*?s=*
Disallow: /*/cgi-bin/*
Disallow: /*/feed/
Disallow: /*/feed/*
Disallow: /*/trackback/
Disallow: /*/trackback/*
Disallow: /*/wp-*
Disallow: /*/xmlrpc.php
Disallow: /*?
disallow: /*?*
Disallow: /archives/
Disallow: /author
Disallow: /cgi-bin/
Disallow: /comments/
Disallow: /comments/feed/
Disallow: /feed/
Disallow: /index.php
Disallow: /login/
Disallow: /wp-*
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-content/cache/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-includes/
Disallow: /wp-rss.php
Disallow: /wp-rss2.php
Disallow: /xmlrpc.php
Disallow: ?wptheme=
Disallow: /readme.html
Disallow: /license.txt
Allow: /wp-content/uploads

User-agent: Mediapartners-Google*
Disallow:
Allow: /

User-agent: Googlebot-Image
Disallow:
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Disallow:
Allow: /

User-agent: Googlebot-Mobile
Disallow:
Allow: /

User-agent: duggmirror
Disallow: /

User-agent: ia_archiver
Disallow: /
share|improve this question
2  
Unfortunately, users won't know what you're trying to accomplish here, so only general opinions could be offered. If you can edit this to a specific concern that you have (e.g., "Is the following correct to do ___",) then it will be reviewed to be reopened. –  dan May 23 at 3:07
    
Thanks for your feedback. I adjusted my question. –  dasickle May 23 at 12:18

1 Answer 1

up vote 1 down vote accepted

I think this one Disallow: /wp-* should be included:

Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-content/cache/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-includes/
Disallow: /wp-rss.php
Disallow: /wp-rss2.php

So you need Disallow: /wp-* only, not all.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.