Timeline for Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?
Current License: CC BY-SA 3.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 9, 2022 at 18:00 | history | tweeted | twitter.com/StackWebmasters/status/1601275500856500230 | ||
| Jan 18, 2015 at 8:12 | comment | added | riffraff | I have some other pages which are served both as .html and .json and behave like normal pages for web and as restful endpoint for our mobile apps, but the one requested by googlebot is not, it's just a dynamic html page. | |
| Jan 17, 2015 at 18:30 | comment | added | Jamin Quimby | Is the .json file a dynamic file served by a restful API | |
| Jan 16, 2015 at 8:32 | history | edited | riffraff | CC BY-SA 3.0 |
update question because one subquestion is solved
|
| Jan 16, 2015 at 8:30 | comment | added | riffraff | @w3d I hadn't thought of that but yes, the ip address does look like googlebot i.e. 66.249.67.87 (aka crawl-66-249-67-87.googlebot.com), I have updated the question. | |
| Jan 15, 2015 at 18:35 | comment | added | Stephen Ostermiller♦ |
I'd never heard of the HTTP_FROM header before this. It appears that Googlebot does set the header and that its presence usually does indicate that it is the real Googlebot because it is rarely spoofed: webmasterworld.com/search_engine_spiders/4422323.htm
|
|
| Jan 15, 2015 at 14:19 | comment | added | MrWhite | Have you verified (by IP) that these requests are from the real Googlebot? | |
| Jan 15, 2015 at 11:56 | history | edited | Simon Hayter | CC BY-SA 3.0 |
minor
|
| Jan 15, 2015 at 10:25 | history | asked | riffraff | CC BY-SA 3.0 |