On one site I own, I recently started seeing Googlebot checking for non-existing URIs: - - [23/Feb/2020:10:18:48 +0100] "GET /robots.txt HTTP/1.1" 404 118 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-" - - [23/Feb/2020:10:18:49 +0100] "GET /admin/install.php HTTP/1.1" 404 181 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"

This would all be well and good, if not for the fact that it has never done so before, this URI never existed (I own the domain for 10+ years) and looks suspiciously like casually scanning for possible security issues. domain name pointer crawl-66-249-76-89.googlebot.com. is also indeed a Googlebot address.

Can anyone shed more light on this?

  • 2
    Maybe it's been linked to from somewhere? – MrWhite Feb 23 '20 at 14:55
  • The link wouldn't necessarily have be on your site. Anybody can create links to non-existent pages on your site. Somebody could even create such links maliciously to get Googlebot to crawl URLs that could hurt your site. – Stephen Ostermiller Nov 20 '20 at 12:09

For other sites (as for me) this is a kind of pretty usual behavior of Googlebot - to crawl non-existing pages.

But don't worry - it will stop it, definitely, with the time. Just care about your non-existing pages have correct status code (404, 410).

Why Googlebot does it? There are some causes:

  • It finds something in your javascripts, what looks similar to paths - and tries to crawl them to maybe find some more (~truffles~) content, greedy pig.
  • It tries to get new content from adding different paths and file names together.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.