0

I'd like to block and unblock following URL formats:

Allow:https://example.com/buyer/#/search/catalog/Lubricating-Oils/Automotive-Lubricants

Dissallow:https://example.com/buyer/#/search/product-description?sku=16087

Due to # in URLs, both are getting allowed or disallowed at a time.

3
  • # is for within page navigation. It is not for producing distinct pages. Think about it, they are the same page. Nov 30, 2022 at 6:53
  • These are SPA pages that is the reason # is present Nov 30, 2022 at 7:56
  • You can't use robots.txt for this because those are client side-only URLs. The only part of that URL that gets crawled is https://example.com/buyer/. See How to disallow hash fragments in robots.txt file Nov 30, 2022 at 9:59

1 Answer 1

1

# is used as a comment character in robots.txt

The # character marks the beginning of a comment.

From Google's Create a robots.txt file

It does not say that # can be escaped. Furthermore

Valid robots.txt lines consists of a field, a colon, and a value. Spaces are optional, but recommended to improve readability. Space at the beginning and at the end of the line is ignored. To include comments, precede your comment with the # character. Keep in mind that everything after the # character will be ignored. The general format is :<#optional-comment>.

How Google interprets the robots.txt specification

The robots.txt file crawlers do not care if you have a SPA or a PWA. I have written a SPA as well. I used params ?name=value for pages. And you can put these in robots.txt.

Not the answer you're looking for? Browse other questions tagged or ask your own question.