1

What is the meaning of `Disallow: /? in a robots.txt file?

User-agent: *  
Disallow: /?*  
4
  • 1
    The statement means all user-agents are disallowed from accessing URLs that starts with a query string at the root directory. It appears that you didn't bother to search the Internet before asking. What is a robots.txt file?
    – Tim R
    Aug 19, 2023 at 8:36
  • 1
    @TimR Please help keep the tone of the comments section welcoming for new users. If you believe a question is without merit or shows no research effort, you can vote it down. Thanks. Aug 19, 2023 at 16:47
  • 1
    Hi there, It means everything that start with question mark in "?" is not allowed to crawl and index. For instance example.com/category/?sort=from1to10 and similar type of URL will be disregarded to crawl. Also these type of query parameter pages can be there on any directory, it will disregard every occurrence. Aug 20, 2023 at 1:39
  • @DeepakMathur, you should make it an answer Aug 20, 2023 at 12:42

1 Answer 1

1

Within a robots.txt file a Disallow rule blocks crawling and Allow rules allow crawling. Bots that respect robots.txt files need to compare the path of the current URL with the paths specified in the Disallow and Allow rules.

The parsing is quite complex and is described in RFC 9309. The matching must be done case sensitive, the most specific match must be used if Allow rules and Disallow rules a equivalent, Allow should be used.

In your example URLs with a path starting with "/?" will be excluded from crawling.

What's noteworthy here is that due to the wildcard Disallow: /?* and Disallow: /? are equivalent, because the path comparison always is a left match.

Here is a table showing the status for some URLs:

URL status
https://example.com/ allowed
https://example.com/? disallowed
https://example.com/?foo disallowed
https://example.com/foo? allowed
https://example.com/foo/? allowed

Note: If you want to block all URLs with a query the rule is insufficient and Disallow: /*? must be used instead.

1
  • 1
    Not sure why the downvote, looks like a comprehensive answer +1
    – MrWhite
    Jan 25 at 0:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.