2

My website has 2 types of URLs

  1. URLs that end with .html -> example.com/example.html
  2. URLs that do not end with .html -> example.com/example

The .html version is the version I want to be indexed. The non .html version is auto generated by the CMS and there is nothing I can do about it technically. I can't noindex the URLs that do not end with .html

That's why I want to block access to URLs that do not end with .html

I know that Disallow: /*.html$ will block all URLs ending with .html. The question is how do I do the opposite?

How do I block all URLs that DO NOT end with .html?

Is there another solution expect robots.txt?

1
  • If the non html versions have identical urls with html, then you should redirect them. Sep 29, 2022 at 4:44

1 Answer 1

3

Short answer: you don't disallow these pages. Disallowing duplicates is not the best practice.

You have rel canonical for it. And your task is trivial from the point of the canonical.

A better solution here, however, would be redirecting to the html version of the page from a non-html version.

It's also worth to say that it's quite debatable whether it makes sense to add non-contributing characters to the url. I prefer Occam's Razor approach where if something doesn't add value in the url, it shouldn't be there, so I would be setting up the redirections the other way around: from .html to the clear url.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.