I want to block robots from crawling URLs that contain the word "tag" on my website. Here are some example URLs:

  1. https://www.example.com/six-senses-fort-barwara-a-glimpse-into-the-spectacular-celebrity-destination-wedding-hotel/
  1. https://www.example.com/tag/six-senses-fort-barwara/

Where ever the "tag" comes in the URL, robots should be blocked from crawling any of the URLs in that subdirectory.


You can "block" the entire /tag directory:

User-agent: *
Disallow: /tag/

Also, consider adding the following to your "tag" pages:

<meta name="robots" content="noindex,nofollow" />

And make sure those pages are also excluded from your sitemap.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy