Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
I want to disallow robots to index some pages of my site. For example, I want to disallow the following pages:
/category/name1/page/1
/category/name1/page/2
/category/etc/page/3
But I want to allow these pages:
/category/name1/
/category/name1/
/category/etc/
How do I express this in robots.txt?
User-agent: *
Disallow: /category/*/page/*
Is that right or not?
To disallow crawling of the URLs ending in page/ then yes, you can use a wildcard like you mentioned. However, there is no need to use a wildcard at the end of the pattern, since robots.txt is prefixing matching by default.
User-agent: *
Disallow: /category/*/page/
That will still allow the crawling of the less specific URLs mentioned.
Note that the wildcard * is an extension of the "standard". Whilst all the main search engine bots support it, not all (good) bots do.