New answers tagged crawl-errors
Your PHP code isn't modifying the robots.txt file. You've merely commented out the link to the page. Try creating a robots.txt file in the main public directory of your website. Then add the following: User-Agent: * Disallow: /terms-and-conditions/ You could also add a noindex meta tag to the file in question: <meta name="robots" ...
If the page is gone, then let it 404 error. If you can, then I suggest doing a 410 gone, but if there are quite a few, then it may not be worth the effort. NEVER 301 redirect a page that no longer exists unless you are redirecting to another relevant page. This should be done where ever possible for the sake of the user. Here is what happens: Using Google ...
You can perform a 301 redirect for the 404 pages to their relevant equivalent on the new site, or just to the home page. You should be more concerned with human visitors. The web crawlers will eventually stop crawling the old pages.
Top 50 recent answers are included