2

I have a new website that has a different product URL from the old website.

URL Old Website

  • https://www.example.com/product/MA-593838554.html
  • https://www.example.com/product/MR-559927541.html
  • etc

Old indexed urls with .html extension still appears in google search. How to block all urls whose extension is .html in robot.txt?

Can I write it like this Disallow: /product/*.html$?

Improve this question
1
  • Officially google and bing both supports $ and * as regex in robot.txt, ideally something like what you suggested would work
    – Manoj Purohit
    Commented Dec 14, 2021 at 7:41

1 Answer 1

Reset to default
1

If you use the disallow directive in your robots.txt file, then Googlebot access to all html pages in the / product / folder will be restricted. Instead you can:

  1. redirect old page to a new page with 301-redirect CMS plugin or .htaccess rule (if your website of the same host):

    RewriteEngine On

Redirect 301 /product/MA-593838554.html /product/MR-559927541.html

or

  1. specify the canonical tag from the old page to the new one in the <head>section</head> of each page. For example, for a page https://www.example.com/product/MA-593838554.html:

    <link rel="canonical" href="https://www.example.com/product/MR-559927541.html"/>

Improve this answer
3
  • can you give me a sample code of the 2 ways you told to me?
    – fikfe
    Commented Dec 13, 2021 at 11:05
  • Your Redirect rule doesn't look right to me. First the page to be redirected and the target are exactly the same. The page to redirect should have only the path and the page to redirect to can have only the path on new version of Apache. I'd expect something more like Redirect 301 /product/MA-593838554.html /product/MA-593838554
    – Stephen Ostermiller ♦
    Commented Dec 13, 2021 at 15:08
  • The new URL may be either an absolute URL beginning with a scheme and hostname, or a URL-path beginning with a slash. I'm edited my answer.
    – Roman Mikhol
    Commented Dec 13, 2021 at 17:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.