2

While analysing a website with the Screaming Frog SEO Sider I've seen a problem of some pages containing duplicate URL directories. To fix this problem I need to know which URLs this affects.

So I have for example this URL

example.com/abc/def/abc/

but maybe also

example.com/xxx/yyy/xxx/

How can I filter URLs with double directories in a URL separated by n directories in between with RegEx?

I've already tried this RegEx \b(\w+)\s+\1\ but it only works if any duplicate words follow each other

Edit:

I asked ChatGPT for RegEx that finds all URLs containing doubled directories like those

  • https://www.example.com/fr-en/abc/xxx/abc/gefhij/
  • https://www.example.com/at-en/def/xxx/yyy/def/xyzabc/
  • https://www.example.com/at-en/def/xxx/yyy/zzz/def/xyzabc/
  • https://www.example.com/at-en/def/def/hscxv/

ChatGPT got me that RegEx

^(https?://[^/]+/)([^/]+/)+[^/]+(/[^/]+)*/\2

This RegEx solely covers the first three cases with directories separated by n other directories, but not the one with directories directly one after the other.

Can anybody help me combine those RegEx to match all for cases?

Improve this question
4
  • 1
    I'm not exactly sure what the regex would be, but the likely cause of this is links that aren't "root relative." That is relative links that don't start with a slash. So on /abc/def/foo.html you might have a link like href=abc/ or href=./abc/ when it would be better to have href=/abc/.
    – Stephen Ostermiller ♦
    Commented Oct 6, 2023 at 16:23
  • Thanks @StephenOstermiller for your input! Would be nuts if it was so easy. I checked the links and yes, they are relative, but with the full path starting from the root and starting with the slash. I guess, the editor just created a duplicate site structure for whatever reason.
    – Robin
    Commented Oct 9, 2023 at 7:47
  • "and yes, they are relative, but with the full path starting from the root and starting with the slash." - If the URL-paths in the HTML source start with a slash then they are "root-relative", not simply "relative".
    – MrWhite
    Commented Oct 10, 2023 at 12:19
  • 2
    My guess is that while most of your links are OK, there are probably a few that are not root-relative. Hopefully the crawler will help you figure out which ones they are.
    – Stephen Ostermiller ♦
    Commented Oct 10, 2023 at 14:26

2 Answers 2

Reset to default
1 
^(https?://[^/]+/)([^/]+/)+[^/]+(/[^/]+)*/\2

This would seem to be overly complex for this particular task. And, like you say, doesn't match the instance when the two "duplicate" directories are next to each other. But neither does it match the first two examples in your question that don't have the language-code prefix in the first path-segment (ie. /abc/def/abc/ and /xxx/yyy/xxx/). In other words, it doesn't match when the first duplicate "directory" appears in the first path-segment.

The following would suffice, to match all 4 examples where a whole path-segment is duplicated anywhere in the URL.

/([^/]+)/(.+/)?\1/

(The / prefix and suffix are part of the regex, they are not delimiters.)

Broken down, the regex matches as follows:

  • /([^/]+) - This matches a whole path-segment and captures the part between the slash delimiters.

  • /(.+/)? - This matches any additional path-segments (or none) that might occur before the "duplicated" path segment.

  • \1 - An internal backreference that matches against the first captured group. This "tests" for a duplicate.

  • / - A literal slash (since your URLs all appear to end in a trailing slash).

This does assume you don't have legitimate path-segments ("directories") of the same name as the hostname. eg. https://www.example.com/at-en/def/www.example.com/hscxv/ would also be caught by this regex. (Assuming the absolute URL is indeed being tested here.)

As mentioned above, this regex should be sufficient for this specific "debugging" task, but it's not particularly efficient due to the backtracking that can potentially occur.

Although, as @StephenOstermiller mentioned in comments, the incorrect use of relative URL-paths in your HTML anchors could also result in "malformed" links of this nature. However, from your second lot of examples, if that was the case then your links would also seem to be missing the language-code prefix, so simply making them root-relative (by including a slash prefix) would not be sufficient.

Improve this answer
0

The Solution thanks to ChatGPT and a lot of trial and error is:

^(https?://[^/]+/)([^/]+/)+([^/]+/)(?:[^/]+/)*\3

Improve this answer
2
  • Did you read/try the "simpler" solution presented in my answer? This solution (from Chat-GPT) does not actually work for the first 2 examples you posted in your question, without the language prefix on the URL-path (ie. /abc/def/abc/ and /xxx/yyy/xxx/). It's also arguably over complex for this use case. It's also a bit odd that it uses a non-capturing subpattern for the 4th group (where it's not needed) but omits this in the first two groups, where it would be beneficial (although the first does not need to be a "group" at all).
    – MrWhite
    Commented Oct 10, 2023 at 18:48
  • I did't received any notification about your answer. Unfortunately I first saw your answer after the page reloaded. So anyway thank you for your contribution. For my case my solution works well even if you're right that is overly complex and yours is way simpler. I tried out both solution against different possible cases like that one example.com/at-en/def/xxx/def/xxx/yyy/yyy/xyzabc/xyzabc even if it is not so likely that the case will occur. Your Regex does not highlight every duplicate directory which on the other hand doesn't matter because this URL is been found anyway
    – Robin
    Commented Oct 11, 2023 at 7:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.