What is a good robots.txt for WP?

Question

What is the "best" setup for robots.txt?
I'm using the following permalink structure in Wordpress: /%category%/%postname%/.

My robots.txt currently looks like this (copied from somewhere a long time ago):

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */comments

1. I want my comments to be indext. So I can remove this, right?
2. Do I want to disallow indexing categories because of my permalinkstructure?
3. An article can have several tags and be in multiple categories. This may cause duplicates in google. How should I work around this?

Would you change anything else here?

Answer 1

The WordPress Codex has an example SEO optimized robots.txt file: http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress#Robots.txt_Optimization

I use the following in the functions.php file of my theme to handle it:

// add to robots.txt
add_action('do_robots', 'roots_robots');

function roots_robots() {
    echo "Disallow: /cgi-bin\n";
    echo "Disallow: /wp-admin\n";
    echo "Disallow: /wp-includes\n";
    echo "Disallow: /wp-content/plugins\n";
    echo "Disallow: /wp-content/cache\n";
    echo "Disallow: /wp-content/themes\n";
    echo "Disallow: /trackback\n";
    echo "Disallow: /feed\n";
    echo "Disallow: /comments\n";
    echo "Disallow: /category/*/*\n";
    echo "Disallow: */trackback\n";
    echo "Disallow: */feed\n";
    echo "Disallow: */comments\n";
    echo "Disallow: /*?*\n";
    echo "Disallow: /*?\n";
    echo "Allow: /wp-content/uploads";
}

Answer 2

1. You should be safe to remove the */comments line if you want to index comments.
2. From what I've experienced, categories are definitely good to index. I would not disallow them.
3. Google should handle it. Check to see if pages you know could be duplicates have the "canonical" tag on them, which identifies the page as a unique resource. If you have duplicate content on these pages you could configure the plugin "All in one SEO pack" to probably take it off those pages: http://wordpress.org/extend/plugins/all-in-one-seo-pack/

Answer 3

I would use this:

User-agent: *
Disllow: /*.zip$
Disllow: /*.doc$
Disllow: /*css$
Disllow: /anyother_download_file$
Disllow: /tag/
Disllow: /category/
Disllow /wp-*

Hope you find this helpful.