2

For some reason Google is listing a series of pages that do not exist on my website such as:

https://www.my_domain.com/index.php/about_us.php

It lists them as 'Duplicate, Google chose different canonical than user' in Search Console.

Google is making up every combination of 'real page' and tacking on another page at the end.

My index.php page is not a folder so why does Google crawl it as if it is a folder with all my pages under it?

3

Googlebot never makes up URLs. It crawls URLs because it finds links to them. If Google is crawling a URL, it must be linked from somewhere on the web.

In this case, one bad link into a page with an extra trailing slash probably caused your site to show additional broken links to Googlebot.

By default, web servers and PHP allow any additional path to be appended to a PHP file and it will show the same content as the original PHP file. So on your server all these URLs probably show the contents of index.php

  • /index.php
  • /index.php/
  • /index.php/foo
  • /index.php/about_us.php

You could fix this by putting a rewrite rule at the top of .htaccess to redirect away from the added path:

RewriteEngine on
RewriteRule (.*\.php).+ /$1 [R=301,L]

Before you do that, you should take a look at the page you serve when the URL is not what you expect. So if somehow a link got created to /index.php/, it seems likely that all the links on the page are broken. You are probably linking with relative URLs like <a href="about_us.php">About Us</a>. From /index.php that will work fine, but from /index.php/ the link doesn't go where you expect.

I recommend changing all your links to "root relative" links that begin with a slash. Doing so prevents problems such as this because the links are relative to your site's domain name, and no longer relative to directory they are in. So I would recommend changing all your links to add a slash at the beginning like <a href="/about_us.php">About Us</a>. With root relative linking, even if Googlebot finds one malformed URL, it isn't going to propogate the problem to every link on the page.

5
  • That makes total sense Stephen. When switching from WordPress to Bootstrap I neglected to verify all the links on the pages. At least one page actually had index.php/another_page as a link and all links were relative, not root relative. Feb 22 '20 at 2:01
  • I tried the Rewrite rule as well and it gave errors until I realized it should be RewriteRule. Once fixed it works marvelously. Feb 23 '20 at 21:47
  • Thank you for finding my mistake I corrected it in the answer. I'm glad it worked for you. Feb 24 '20 at 0:57
  • One thing that still troubles me, however is that Google Search Console still lists these pages as 'Excluded' and when I use the 'Test Page' function it still gives me the option to request that Google index it and the test response is HTTP/1.1 200 OK. Feb 25 '20 at 1:10
  • It is appropriate that Google excludes them from the search index. They are accidental URLs that show the same content as a different preferred URL. You shouldn't request that they be indexed. I think there is a difference between "Test Page" and "Live Test". A test that isn't "live" may use data from the last crawl, which would be from before you fixed the issue. Feb 25 '20 at 12:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.