Skip to main content
3 of 4
added 6 characters in body
Robin
  • 21
  • 2

Regex / Regular Expression for finding duplicate URL directories in Screaming Frog SEO Spider

While analysing a website with the Screaming Frog SEO Sider I've seen a problem of some pages containing duplicate URL directories. To fix this problem I need to know which URLs this affects.

So I have for example this URL
example.com/abc/def/abc/

but maybe also
example.com/xxx/yyy/xxx/

How can I filter URLs with double directories in a URL devided by n directories in between with RegEx?

I've already tried this RegEx \b(\w+)\s+\1\ but it only works if any duplicate words follow each other

Edit:

I asked ChatGPT for RegEx that finds all URLs containing doubled directories like those https://www.example.com/fr-en/abc/xxx/abc/gefhij/
https://www.example.com/at-en/def/xxx/yyy/def/xyzabc/
https://www.example.com/at-en/def/xxx/yyy/zzz/def/xyzabc/
https://www.example.com/at-en/def/def/hscxv/

ChatGPT got me that RegEx
^(https?://[^/]+/)([^/]+/)+[^/]+(/[^/]+)*/\2

This RegEx solely covers the first three cases with directories devided by n other directories, but not the one with directories directly one after the other.

Can anybody help me combine those RegEx to match all for cases?

Robin
  • 21
  • 2