I have a similar URL in my website that I want to stop all robots from crawling /review/dir/dir/dir/dir/#review-form

The rules I've tried are:

  • Disallow:/review/*review-form$
  • Disallow:/*\#review-form$
  • Disallow:review-form$
  • Disallow:/#review-form

The bottom rule works but also blocks the URL without the hash fragment because the # symbol starts a comment.

How can I disallow a URL that has a hash fragment in the robots.txt file?

  • 3
    Browsers and crawlers never send the fragment identifier (the portion of the URL after the #) to the server. It doesn't make sense to try to disallow that, because it will never be fetched. – Stephen Ostermiller Nov 6 '17 at 20:27
  • @GrapeSoda Do you have evidence to suggest that URLs with frag ids are being requested by bots? Does your page deliver different content based on the frag id? – MrWhite Nov 7 '17 at 12:40

SOURCE Comments can be included at any location in the file using the "#" character.

You should be able to get around the issue by using URL Encoding and using the code %23, like so:

  • Disallow: /%23your-url/
  • 3
    %23 is not the same as # in the URL. The hash separates the fragment identifier section, %23 is needed for a hash in other parts of the URL and doesn't start the fragment. – Stephen Ostermiller Nov 6 '17 at 20:25
  • Oh, I see now. So better to use /#!/reviews then use ?_escaped_fragment_=reviews? in the robots.txt? – Simon Hayter Nov 6 '17 at 21:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.