User-agent: *
Allow: /sitemapindex.xml
Allow: /post_google_news.xml
Allow: /news/
Allow: /category/news/
Disallow: /

I want to exclude everything from being indexed by Google except for example.com/sitemapindex.xml, example.com/post_google_news.xml, example.com/news/anything and example.com/category/news/anything/anything etc.

closed as off-topic by Stephen Ostermiller Aug 30 at 23:34

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions that are specific to one website are discouraged because they are unlikely to help future visitors. Examples include individual website reviews for SEO purposes and general website troubleshooting. See the How to Ask page for help improving this question." – Stephen Ostermiller
If this question can be reworded to fit the rules in the help center, please edit the question.


Yes, this robots.txt file will work. Only comment here is, your /news/ and /category/news/ seem like they might be two different file paths to the same content? If so, I'm assuming you've already selected your canonical URL's and noindexed anything that shouldn't show up in search results using meta tags; in that case, feel free to include only the file path you want to appear in the SERP's, and take out the other one.

Not the answer you're looking for? Browse other questions tagged or ask your own question.