4

I have a similar URL in my website that I want to stop all robots from crawling /review/dir/dir/dir/dir/#review-form

The rules I've tried are:

  • Disallow:/review/*review-form$
  • Disallow:/*\#review-form$
  • Disallow:review-form$
  • Disallow:/#review-form

The bottom rule works but also blocks the URL without the hash fragment because the # symbol starts a comment.

How can I disallow a URL that has a hash fragment in the robots.txt file?

2
  • 4
    Browsers and crawlers never send the fragment identifier (the portion of the URL after the #) to the server. It doesn't make sense to try to disallow that, because it will never be fetched. Commented Nov 6, 2017 at 20:27
  • @GrapeSoda Do you have evidence to suggest that URLs with frag ids are being requested by bots? Does your page deliver different content based on the frag id?
    – MrWhite
    Commented Nov 7, 2017 at 12:40

2 Answers 2

1

The fragment is not part of the HTTP request, so it is unlikely to work. A browser when you give it a URL sends a request for the path and query parameters. Then, when the response is obtained it jumps to the specified fragment. A robot would have to do the same, request the URL without the fragment. Although I am not sure about the latter, I did work on Chrome and so I am certain about the former.

0

SOURCE Comments can be included at any location in the file using the "#" character.

You should be able to get around the issue by using URL Encoding and using the code %23, like so:

  • Disallow: /%23your-url/
2
  • 3
    %23 is not the same as # in the URL. The hash separates the fragment identifier section, %23 is needed for a hash in other parts of the URL and doesn't start the fragment. Commented Nov 6, 2017 at 20:25
  • Oh, I see now. So better to use /#!/reviews then use ?_escaped_fragment_=reviews? in the robots.txt? Commented Nov 6, 2017 at 21:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.