Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e.

  • REQUEST_PATH : /foo/bar.json
  • HTTP_HOST : www.myapp.com
  • HTTP_FROM : googlebot(at)googlebot.com
  • HTTP_REFERER : android.app.1.8.2 (the one we use in our android app)
  • HTTP_USER_AGENT : Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

www.myapp.com/foo/bar is a valid page but there is nothing being served as json.

As far as I can tell, the invalid link with the .json extension does not appear anywhere (and in fact, I cannot find the string indexed in google at all).

What could be causing this?

Is it possible google is looking at json data in our android app (where the link without json might legitimately appear) and crawling it?

Is there any way I could avoid this, to avoid my logs being spammed with spurious requests?

Is it even likely that this is in fact google rather than something else?The request seems like a legitimate googlebot request based on the IP address.

  • 2
    Have you verified (by IP) that these requests are from the real Googlebot? – MrWhite Jan 15 '15 at 14:19
  • I'd never heard of the HTTP_FROM header before this. It appears that Googlebot does set the header and that its presence usually does indicate that it is the real Googlebot because it is rarely spoofed: webmasterworld.com/search_engine_spiders/4422323.htm – Stephen Ostermiller Jan 15 '15 at 18:35
  • @w3d I hadn't thought of that but yes, the ip address does look like googlebot i.e. (aka crawl-66-249-67-87.googlebot.com), I have updated the question. – riffraff Jan 16 '15 at 8:30
  • Is the .json file a dynamic file served by a restful API – Jamin Quimby Jan 17 '15 at 18:30
  • I have some other pages which are served both as .html and .json and behave like normal pages for web and as restful endpoint for our mobile apps, but the one requested by googlebot is not, it's just a dynamic html page. – riffraff Jan 18 '15 at 8:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.