2

On one site I own, I recently started seeing Googlebot checking for non-existing URIs:

66.249.76.89 - - [23/Feb/2020:10:18:48 +0100] "GET /robots.txt HTTP/1.1" 404 118 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
66.249.76.87 - - [23/Feb/2020:10:18:49 +0100] "GET /admin/install.php HTTP/1.1" 404 181 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"

This would all be well and good, if not for the fact that it has never done so before, this URI never existed (I own the domain for 10+ years) and looks suspiciously like casually scanning for possible security issues.

89.76.249.66.in-addr.arpa domain name pointer crawl-66-249-76-89.googlebot.com. is also indeed a Googlebot address.

Can anyone shed more light on this?

Improve this question
2
  • 2
    Maybe it's been linked to from somewhere?
    – MrWhite
    Commented Feb 23, 2020 at 14:55
  • The link wouldn't necessarily have be on your site. Anybody can create links to non-existent pages on your site. Somebody could even create such links maliciously to get Googlebot to crawl URLs that could hurt your site.
    – Stephen Ostermiller ♦
    Commented Nov 20, 2020 at 12:09

1 Answer 1

Reset to default
1

For other sites (as for me) this is a kind of pretty usual behavior of Googlebot - to crawl non-existing pages.

But don't worry - it will stop it, definitely, with the time. Just care about your non-existing pages have correct status code (404, 410).

Why Googlebot does it? There are some causes:

  • It finds something in your javascripts, what looks similar to paths - and tries to crawl them to maybe find some more (~truffles~) content, greedy pig.
  • It tries to get new content from adding different paths and file names together.
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.