3

I've been trying to clean up crawl errors on my company's website. In one menu, google bot is trying to crawl every link with an extra-base URL attached. These pages (perhaps obviously) don't exist and return a "400 Bad Request" error.

GSC screenshot showing 400 errors

The site is built on PHP and Apache. The URL's are written dynamically by PHP, but there are also some 301 redirects associated with the pages in question.

Has anyone seen and or solved this before? Will it affect my SEO?

  • 1
    Are you sure Googlebot is crawling https://www.example.com/https://www.example.com/other/things and not https://www.example.com/www.example.com/other/things? The later is easy to explain, the former is not. – MrWhite May 9 '18 at 18:16
  • 1
    imgur.com/zSszPue – Anne Elise Parmentier May 9 '18 at 18:53
  • Yes is is crawling example.com/https://www.example.com/other/things. It also says it is linked from example.com/https://www.example.com/other/stuff (which seems odd since example.com/https://www.example.com/other/stuff also returns a 400 error) – Anne Elise Parmentier May 9 '18 at 18:56
  • That would certainly seem to be a strange one! Having had a look at your site, I can't see any problems. Maybe there was a problem which is now fixed? Maybe Googlebot is seeing something different to users (although faking the user-agent doesn't reveal anything unusual)? Does the "Fetch and Render" tool in GSC show anything unusual? Your pages appear to be indexed OK - including the "correct" version of the URLs you appear to be referring to? – MrWhite May 9 '18 at 22:26
  • 1
    "In one menu" - is this a menu on your site? or in GSC? – MrWhite May 9 '18 at 23:10
3

Why is Google still indexing those links?

It's because Google Bot found those link on your site at some point in time and that's why it indexed it. Google Bot doesn't make up URLs so it must of found it on your website. If Google indexed something once, it will come back and revisit it, even if the destination resource doesn't exists or the original link to it doesn't exist on your website anymore. My guess is that it was a programming mistake because it's easy to make that mistake, especially if your have do URLs in PHP.

Does this affect SEO?

Not really, links resulting in 404 errors don't affect your SEO. However, I would be much more concerned about the 70+ of your website ZOMBIES pages. Basically, Google crawler started indexing your website's search controller (yourwebsite.com/search). If you type into Google "site:yourwebsite.com", it will show you all indexed pages for that domain. See your Zombie pages here.

How to fix it?

  • Redirect these links to throw error 410 (Gone) - this tells Google to not revisit that URL anymore

  • Mark the links as fixed in the Google Search Console

  • 1
    It doesn't look like the pages give 404 errors. In the screenshot from the question, some of the pages are listed as "linked from.". To me that indicates the site is somehow serving pages for URLs like that that then link to other pages with similar URL structure. I'd recommend fixing that issue. – Stephen Ostermiller May 14 '18 at 9:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.