3

I want to disallow robots to index some pages of my site. For example, I want to disallow the following pages:

/category/name1/page/1
/category/name1/page/2
/category/etc/page/3

But I want to allow these pages:

/category/name1/
/category/name1/
/category/etc/

How do I express this in robots.txt?

User-agent: *
Disallow: /category/*/page/*

Is that right or not?

1

To disallow crawling of the URLs ending in page/ then yes, you can use a wildcard like you mentioned. However, there is no need to use a wildcard at the end of the pattern, since robots.txt is prefixing matching by default.

User-agent: *
Disallow: /category/*/page/

That will still allow the crawling of the less specific URLs mentioned.

Note that the wildcard * is an extension of the "standard". Whilst all the main search engine bots support it, not all (good) bots do.

  • Can't i use wildcard? Because I have thousands pages and category names so i must write each page? – Muhammad Dyas Yaskur Dec 2 '15 at 23:06
  • Sorry, yes you can use a wildcard - just realised the significance of the wildcard in your question! I've updated my answer. – MrWhite Dec 2 '15 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.