What is the "best" setup for robots.txt?
I'm using the following permalink structure in Wordpress:
My robots.txt currently looks like this (copied from somewhere a long time ago):
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */comments
- I want my comments to be indext. So I can remove this, right?
- Do I want to disallow indexing categories because of my permalinkstructure?
- An article can have several tags and be in multiple categories. This may cause duplicates in google. How should I work around this?
Would you change anything else here?