2

I found a loads of examples where in robots.txt were disallowed urls with similar approaches:

Disallow: /*?dir=*&order=*&p=*

Where every path with specified parameters will be blocked.

What if I want to block an urls with a specified path? Lets say I want to block: /Home/SomeBlockedPath?dir=false&order=desc&p=12 and have allowed: /Home/AllowedPath?dir=false&order=desc&p=3. Is this approach below fine?

Disallow: /Home/SomeBlockedPath?dir=*&order=*&p=*

3

All right, I found an answer in docs for how to use robots.txt:

URL matching based on path values Robots.txt Specifications

This should be fine:

Disallow: /Home/SomeBlockedPath

5
  • That looks a lot better. The major search engines support the wildcards in robots.txt rules, but most other crawlers don't. Avoid wildcards by using simple prefix rules like this when possible. Mar 17 at 10:03
  • 1
    Using multiple wildcards is problematic even for bots that support wildcards. Your original rule disallows specific parameters used in a particular order. But change the order of the parameters and search engines would be able to crawl. Mar 17 at 10:04
  • @StephenOstermiller Do you think, that Disallow: /Home/SomeBlockedPath is better than Disallow: /Home/SomeBlockedPath* in my case?
    – bakunet
    Mar 17 at 10:17
  • 1
    Without the asterisk is better. The two rules are the same for the bots that support wildcards, but there are a lot of bots that won't understand the rule with the wildcard. Mar 17 at 11:36
  • Great, thank you for comments. SEO is still a black magic to me ;)
    – bakunet
    Mar 17 at 11:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.