3

in our serverlogs there are thousands of requests (50-100k) from googlebot every day for urls like /2487763877595434670.htm (as I can see always 19 random digits with .htm at the end)

First the bot requests http:// url that is redirected to https:// version than we block the request with an 404.

Can someone tell me how to find out why googlebot crawls these urls that have never existed? And of course how to stop the bot to waste it´s crawl budget on those useless urls?

I looked for such requests on different domains and only get these requests on our oldest domains (older than 20 years).

Improve this question
2
  • Have you looked at the crawl report in Google Search Console? Google should be showing these URLs in there and telling you where it is finding them.
    – Stephen Ostermiller ♦
    Commented Jul 19, 2019 at 17:23
  • If I check the urls via URL inspection in Search Console I get "URL is unknown to Google" and thatswhy the referring page is empty.
    – wsm
    Commented Jul 22, 2019 at 10:57

1 Answer 1

Reset to default
3

There is few scenarios to consider.

  1. Check your sitemap files to find if you submit any of these URL's to crawl by Google.
  2. Check in the google search console for failed crawl attempts and verify these requests are originated from Google bot.
  3. Get the IP address from the server logs and check few IP's for address details to verify these requests are originated from Google servers.

If you found above mentioned URL's in sitemap files you need to update the sitemap and tell the google that these URL's are no longer exists.

If you found out that these requests are not originated from Google servers this may be some attach or other application trying to query your website using google bot as the user agent.

Improve this answer
5
  • Thanks for your reply. 1. These URLs are not in sitemaps, as there are thousands of new useless urls every day, no one would put that many in sitemaps :) 2./3. The requests are originated from Google servers, IPs are from 66.249.76.xx cluster and via ip lookup that are all Googlebot / Google data centers.
    – wsm
    Commented Jul 22, 2019 at 9:52
  • maybe session ID's in URL ?
    – Shyamin Ayesh
    Commented Jul 22, 2019 at 11:06
  • That the 19 random digits could be a session ID? No because session IDs would be a mixture of numbers and letters and would start with 'SID='
    – wsm
    Commented Jul 22, 2019 at 12:44
  • not sure. but a wild guess. is Google following from do follow links placed on other website maybe ?
    – Shyamin Ayesh
    Commented Jul 22, 2019 at 14:09
  • Checked now links on search console and ahrefs but can´t find any that look only a little bit similiar to that the bot crawls.
    – wsm
    Commented Jul 23, 2019 at 9:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.