User-agent: *
Allow: /sitemapindex.xml
Allow: /post_google_news.xml
Allow: /news/
Allow: /category/news/
Disallow: /

I want to exclude everything from being indexed by Google except for example.com/sitemapindex.xml, example.com/post_google_news.xml, example.com/news/anything and example.com/category/news/anything/anything etc.


Yes, this robots.txt file will work. Only comment here is, your /news/ and /category/news/ seem like they might be two different file paths to the same content? If so, I'm assuming you've already selected your canonical URL's and noindexed anything that shouldn't show up in search results using meta tags; in that case, feel free to include only the file path you want to appear in the SERP's, and take out the other one.

Not the answer you're looking for? Browse other questions tagged or ask your own question.