Skip to main content
Included the "main bots" and reference
Source Link
MrWhite
  • 43.1k
  • 4
  • 50
  • 90

robots.txt is prefix matching, so a rule like Disallow: /?b=9 will block all URLs that start /?b=9. Your URLs start /shopp... so they are not blocked.

However, you can use a * (wildcard - 0 or more instances of any character) to represent the first part of the URL. This is an addition to the "standard", but the main search engine bots ("Google, Bing, Yahoo, and Ask") support it:

Disallow /*/?b=9

The above should block /shopping/books/?b=9 and /<anything>/?b=9.

Reference:
https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#url-matching-based-on-path-values

robots.txt is prefix matching, so a rule like Disallow: /?b=9 will block all URLs that start /?b=9. Your URLs start /shopp... so they are not blocked.

However, you can use a * (wildcard - 0 or more instances of any character) to represent the first part of the URL. This is an addition to the "standard", but the main search engine bots support it:

Disallow /*/?b=9

The above should block /shopping/books/?b=9 and /<anything>/?b=9.

robots.txt is prefix matching, so a rule like Disallow: /?b=9 will block all URLs that start /?b=9. Your URLs start /shopp... so they are not blocked.

However, you can use a * (wildcard - 0 or more instances of any character) to represent the first part of the URL. This is an addition to the "standard", but the main search engine bots ("Google, Bing, Yahoo, and Ask") support it:

Disallow /*/?b=9

The above should block /shopping/books/?b=9 and /<anything>/?b=9.

Reference:
https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#url-matching-based-on-path-values

added 39 characters in body
Source Link
MrWhite
  • 43.1k
  • 4
  • 50
  • 90

robots.txt is prefix matching, so a rule like Disallow: /?b=9 will block all URLs that start /?b=9. Your URLs start /shopp... so they are not blocked.

However, you can use a * (wildcard - 0 or more instances of any character) to represent the first part of the URL. This is an addition to the "standard", but the main search engine bots support it:

Disallow /*/?b=9

The above should block /shopping/books/?b=9 and /<anything>/?b=9.

robots.txt is prefix matching, so a rule like Disallow: /?b=9 will block all URLs that start /?b=9. Your URLs start /shopp... so they are not blocked.

However, you can use a * (wildcard) to represent the first part of the URL. This is an addition to the "standard", but the main search engine bots support it:

Disallow /*/?b=9

The above should block /shopping/books/?b=9 and /<anything>/?b=9.

robots.txt is prefix matching, so a rule like Disallow: /?b=9 will block all URLs that start /?b=9. Your URLs start /shopp... so they are not blocked.

However, you can use a * (wildcard - 0 or more instances of any character) to represent the first part of the URL. This is an addition to the "standard", but the main search engine bots support it:

Disallow /*/?b=9

The above should block /shopping/books/?b=9 and /<anything>/?b=9.

Source Link
MrWhite
  • 43.1k
  • 4
  • 50
  • 90

robots.txt is prefix matching, so a rule like Disallow: /?b=9 will block all URLs that start /?b=9. Your URLs start /shopp... so they are not blocked.

However, you can use a * (wildcard) to represent the first part of the URL. This is an addition to the "standard", but the main search engine bots support it:

Disallow /*/?b=9

The above should block /shopping/books/?b=9 and /<anything>/?b=9.