What is the meaning of `Disallow: /? in a robots.txt file?
User-agent: *
Disallow: /?*
-
1The statement means all user-agents are disallowed from accessing URLs that starts with a query string at the root directory. It appears that you didn't bother to search the Internet before asking. What is a robots.txt file?– Tim RCommented Aug 19, 2023 at 8:36
-
1@TimR Please help keep the tone of the comments section welcoming for new users. If you believe a question is without merit or shows no research effort, you can vote it down. Thanks.– Maximillian Laumeister ♦Commented Aug 19, 2023 at 16:47
-
1Hi there, It means everything that start with question mark in "?" is not allowed to crawl and index. For instance example.com/category/?sort=from1to10 and similar type of URL will be disregarded to crawl. Also these type of query parameter pages can be there on any directory, it will disregard every occurrence.– Deepak MathurCommented Aug 20, 2023 at 1:39
-
1@DeepakMathur, you should make it an answer– Rohit GuptaCommented Aug 20, 2023 at 12:42
Add a comment
|
1 Answer
Within a robots.txt file a Disallow rule blocks crawling and Allow rules allow crawling. Bots that respect robots.txt files need to compare the path of the current URL with the paths specified in the Disallow and Allow rules.
The parsing is quite complex and is described in RFC 9309. The matching must be done case sensitive, the most specific match must be used if Allow rules and Disallow rules a equivalent, Allow should be used.
In your example URLs with a path starting with "/?" will be excluded from crawling.
What's noteworthy here is that due to the wildcard Disallow: /?* and Disallow: /? are equivalent, because the path comparison always is a left match.
Here is a table showing the status for some URLs:
| URL | status |
|---|---|
https://example.com/ |
allowed |
https://example.com/? |
disallowed |
https://example.com/?foo |
disallowed |
https://example.com/foo? |
allowed |
https://example.com/foo/? |
allowed |
Note: If you want to block all URLs with a query the rule is insufficient and Disallow: /*? must be used instead.
-
1