How to fix 404 crawl error of a website

Question

I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below

+top_folder
     -config.php 
     +includes_folder
       -nav.php  (conatins links to all pages in my website)
     +courses_folder
       -course1.php
       -course2.php
       -nav.php

config.php

<?php 
    $main_url = "http://www.myapp.com/";
?>

nav.php file

 <ul>
      <li><a href="<?=$main_url?>courses_folder/course1.php">Course 1</li>
      <li><a href="<?=$main_url?>courses_folder/course2.php">Course 2</li>
  </ul>

So, I am including the top_folder/includes_folder/nav.php file in all pages. But I also placed nav.php in other location: top_folder/courses_folder/nav.php and I am not including this file in any file of my website.

$main_url is only defined in config.php, which I am including in all files, Not in nav.php. So the $main_url is being displayed as blank in crawled page i.e. top_folder/courses_folder/nav.php

Google has crawled top_folder/courses_folder/nav.php page also. The links are not proper bacause $main_url is null in this case.

So It's been a 7 months since i done this changes.

Is it a good idea to just include the config.php in top_folder/courses_folder/nav.php file

OR

Do i need to create the files and folders (As links in top_folder/courses_folder/nav.php file).

OR

Do i need to disallow the links (robots.txt) in top_folder/courses_folder/nav.php file

Among the above, Which is the recommended solution to solve the 404 issues without affecting the SEO.

Answer 1

Google has crawled top_folder/courses_folder/nav.php page also. The links are not proper bacause $main_url is null in this case.

If I understand correctly then the 404's have resulted from Google crawling a page that it shouldn't have. And you see the 404's in Google Webmaster Tools? People are not linking to these incorrect URLs and these do not appear in Google Search (they can't since they don't exist).

In this case, there is nothing you really need to do in my opinion. The 404's are not bad and are not affecting your search engine ranking. A 404 is a correct response.

However, you should prevent Google from crawling your /includes_folder using robots.txt. This will, in time, resolve your 404's, since Google will no longer be able to crawl your nav.php file.

You seem to be concentrating too much on the 404's themselves, rather than what is causing them.

Answer 2

First, it seems like there is nothing to worry about except for crowding in your reports. So, SEO is probably not affected but it would be nice to clean up after yourself.

What you should have done and can still do is use / instead of <?=$main_url?> in nav.php. Also it should never have been crawled since it is not a page AFAIK. If you are including it an iframe, which would explain how Google found it, then you can probably replace that by a PHP include.

This will not remove the errors from the Google Webmaster Tools because Google already thinks that nav.php is a page and remembers the links there. They will eventually disappear but it can take months. To speed things, block the incorrect URLs using robots.txt so that Google stops requesting them.

Several days ago a jerk (sorry for the technical term) created 16,000 bogus links to an nonexistent page on my site and they were reported for 11 consecutive days in GWMT. I have used robots.txt to block Google from requesting a page that is not there and the number of 404s reported has been diminishing by roughly 1000 per day since, so I can vouch for it working at least in one case.