In the Google Search Console, under Indexing's "Duplicate without user-selected canonical" error section - there are many links that are being crawled that are not supported.

I have recently updated my canonical link generation to properly support internationalization, but there are only a handful of languages that I support. Others, like ru, ko, or id are showing up in the list of crawled URLs without me ever requesting it. They are not in the sitemap.

Is there anything I can do to remove these from the graphs? Ideally without updating robots.txt.

enter image description here

  • 1
    What HTTP status and content is your site returning when a language code you don't support his requested? Aug 8, 2023 at 20:51
  • It ignores the parameter and renders the default, en-US page. Status 200 OK
    – James L.
    Aug 8, 2023 at 20:53
  • 1
    It seems, such requests can arise due to some visitors browsers' language preferences. You can use such 'error requests' to analyze viewers of your site.
    – Tom Newton
    Aug 9, 2023 at 8:28

1 Answer 1


Change your site to return a "404 Not Found" status when an unsupported language code is requested.

Googlebot still may crawl the occasional unsupported language parameter, but when it does so, your site won't then link to lots of other such pages. In the end, Googlebot will end up crawling a lot less bad URLs.

  • Would it have a similar impact if I did a JS/Angular redirect in the front end to the 404 page instead of a 404 server error?
    – James L.
    Aug 9, 2023 at 21:02
  • 1
    Yes preventing the page from getting rendered and r redirecting to a 404 is equivalent Aug 9, 2023 at 22:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.