2

I have a website with around 100K URLs. I want to disallow crawling for all URLs that have an ID with this pattern:

www.example.com/node/sport/category/id

but not those without the ID:

www.example.com/node/sport/category/

How can I approach this in a robots.txt?

UPDATE:
the ID are some numbers like /12343/ or /12345/ etc

Improve this question
2
  • 1
    Which parts of these URLs are placeholders? i.e., does the URL actually contain "category" or is this a placeholder? And if so, how many categories are there? And what is the structure of "id"?
    – unor
    Commented May 26, 2019 at 16:28
  • 1
    Do you specifically want to prevent crawling, or would it be sufficient to just prevent indexing? The later you could do with a few directives in .htaccess to inject an X-Robots-Tag: noindex, nofollow HTTP response header when an appropriate URL (matched by regex) is requested.
    – MrWhite
    Commented May 26, 2019 at 17:16

2 Answers 2

Reset to default
1

You don't have a lot of great options. robots.txt rules are usually prefix matching.

One option would be to change all of your URLs that end in an id so that they have a common prefix: /node/sport/category/ would get crawled but /private/node/sport/category/id would not because you would add Disallow: /private/ to robots.txt.

The major search engine bots (Google and Bing) do support pattern matching in robots.txt files. Unfortunately, they don't support regular expression matching, so you are limited much less powerful globs. Even the globs won't be supported by most other crawlers, so this will only work for major search engine bots, not for all crawlers. I think the best you can do is disallow URLs ending in digits with 10 different rules:

Disallow: *0$
Disallow: *1$
Disallow: *2$
Disallow: *3$
Disallow: *4$
Disallow: *5$
Disallow: *6$
Disallow: *7$
Disallow: *8$
Disallow: *9$

Another option is to allow those URLs to be crawled, but prevent them from being indexed. Instead of changing your robots.txt file, you could add a meta tag to those documents so that search engines won't index them after they get crawled:

<meta name="robots" content="noindex">
Improve this answer
3
  • 1
    You could perhaps be more specific and only block a minimum path depth, eg. Disallow: /*/*/*/*0$ etc. (to match the /<node>/<sport>/<category>/<id> path depth).
    – MrWhite
    Commented May 26, 2019 at 17:52
  • I will need to add one entry for each of the unitary numbers? 1,2,3,4, etc? like/<node>/<sport>/<category>/*0$ then the same for 1, 2, 3, etc? that would work for 39383 or 1238437? thanks for your help
    – Diego Wais
    Commented May 27, 2019 at 16:33
  • *0$ means ends with zero. So it would be fine for IDs 10, 200, 34760, etc. Ten rules should cover all possible IDs.
    – Stephen Ostermiller ♦
    Commented May 27, 2019 at 18:22
-1

To achieve this your robots.txt can contain the following lines

User-agent: *
Disallow: /*/*/*/id

You can also test this in the robots.txt checker in the old search console

Improve this answer
3
  • 1
    I think "id" is standing in for some digits (ie a number). This isn't going to work.
    – Stephen Ostermiller ♦
    Commented May 26, 2019 at 9:52
  • Yes if the ID means a digit then this wont work
    – Shahzeb Qureshi
    Commented May 26, 2019 at 12:19
  • 1
    Yes... the ID are some numbers like /12343/ or /12345/ etc etc
    – Diego Wais
    Commented May 26, 2019 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.