Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

What is the "best" setup for robots.txt?
I'm using the following permalink structure in Wordpress: /%category%/%postname%/.

My robots.txt currently looks like this (copied from somewhere a long time ago):

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */comments
  1. I want my comments to be indext. So I can remove this, right?
  2. Do I want to disallow indexing categories because of my permalinkstructure?
  3. An article can have several tags and be in multiple categories. This may cause duplicates in google. How should I work around this?

Would you change anything else here?

share|improve this question
5  
You'll probably get a better response at wordpress.stackexchange.com –  John Conde Dec 13 '10 at 14:35
add comment

3 Answers 3

The WordPress Codex has an example SEO optimized robots.txt file: http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress#Robots.txt_Optimization

I use the following in the functions.php file of my theme to handle it:

// add to robots.txt
add_action('do_robots', 'roots_robots');

function roots_robots() {
    echo "Disallow: /cgi-bin\n";
    echo "Disallow: /wp-admin\n";
    echo "Disallow: /wp-includes\n";
    echo "Disallow: /wp-content/plugins\n";
    echo "Disallow: /wp-content/cache\n";
    echo "Disallow: /wp-content/themes\n";
    echo "Disallow: /trackback\n";
    echo "Disallow: /feed\n";
    echo "Disallow: /comments\n";
    echo "Disallow: /category/*/*\n";
    echo "Disallow: */trackback\n";
    echo "Disallow: */feed\n";
    echo "Disallow: */comments\n";
    echo "Disallow: /*?*\n";
    echo "Disallow: /*?\n";
    echo "Allow: /wp-content/uploads";
}
share|improve this answer
add comment
  1. You should be safe to remove the */comments line if you want to index comments.
  2. From what I've experienced, categories are definitely good to index. I would not disallow them.
  3. Google should handle it. Check to see if pages you know could be duplicates have the "canonical" tag on them, which identifies the page as a unique resource. If you have duplicate content on these pages you could configure the plugin "All in one SEO pack" to probably take it off those pages: http://wordpress.org/extend/plugins/all-in-one-seo-pack/
share|improve this answer
add comment

I would use this:

User-agent: *
Disllow: /*.zip$
Disllow: /*.doc$
Disllow: /*css$
Disllow: /anyother_download_file$
Disllow: /tag/
Disllow: /category/
Disllow /wp-*

Hope you find this helpful.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.