2

I have a new website that has a different product URL from the old website.

URL Old Website

  • https://www.example.com/product/MA-593838554.html
  • https://www.example.com/product/MR-559927541.html
  • etc

Old indexed urls with .html extension still appears in google search. How to block all urls whose extension is .html in robot.txt?

Can I write it like this Disallow: /product/*.html$?

1
  • Officially google and bing both supports $ and * as regex in robot.txt, ideally something like what you suggested would work Dec 14, 2021 at 7:41

1 Answer 1

1

If you use the disallow directive in your robots.txt file, then Googlebot access to all html pages in the / product / folder will be restricted. Instead you can:

  1. redirect old page to a new page with 301-redirect CMS plugin or .htaccess rule (if your website of the same host):

    RewriteEngine On
    
    Redirect 301 /product/MA-593838554.html /product/MR-559927541.html
    

or

  1. specify the canonical tag from the old page to the new one in the <head>section</head> of each page. For example, for a page https://www.example.com/product/MA-593838554.html:

    <link rel="canonical" href="https://www.example.com/product/MR-559927541.html"/>

3
  • can you give me a sample code of the 2 ways you told to me?
    – fikfe
    Dec 13, 2021 at 11:05
  • Your Redirect rule doesn't look right to me. First the page to be redirected and the target are exactly the same. The page to redirect should have only the path and the page to redirect to can have only the path on new version of Apache. I'd expect something more like Redirect 301 /product/MA-593838554.html /product/MA-593838554 Dec 13, 2021 at 15:08
  • The new URL may be either an absolute URL beginning with a scheme and hostname, or a URL-path beginning with a slash. I'm edited my answer.
    – skill
    Dec 13, 2021 at 17:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.