Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e.
- REQUEST_PATH : /foo/bar.json
- HTTP_HOST : www.myapp.com
- HTTP_FROM : googlebot(at)googlebot.com
- HTTP_REFERER : android.app.1.8.2 (the one we use in our android app)
- HTTP_USER_AGENT : Mozilla/5.0 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html)
www.myapp.com/foo/bar is a valid page but there is nothing being served as json.
As far as I can tell, the invalid link with the .json extension does not appear anywhere (and in fact, I cannot find the string indexed in google at all).
What could be causing this?
Is it possible google is looking at json data in our android app (where the link without json might legitimately appear) and crawling it?
Is there any way I could avoid this, to avoid my logs being spammed with spurious requests?
Is it even likely that this is in fact google rather than something else?The request seems like a legitimate googlebot request based on the IP address.
HTTP_FROMheader before this. It appears that Googlebot does set the header and that its presence usually does indicate that it is the real Googlebot because it is rarely spoofed: webmasterworld.com/search_engine_spiders/4422323.htm