Googlebot continues to crawl mp4 files in the format www.example.com/video/12345.mp4?md5=2029zanlkdain312on&expires=1523327783 which has obviously expired (only valid timestamp+30minutes, after which 410 response is given), and causes a crawl anomaly on the parent html url. Some of the timestamps we are seeing are almost 2 years in the past. Strange thing is that google uses a regular useragent ("Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) and not Googlebot-video (Googlebot-Video/1.0). Our sitemap contains direct video urls without md5 and expires parameters as we only allow rdns googlebot verified hosts to fetch our videos without secure_link, and those videos are downloaded fully with Googlebot-video useragent.

We are trying to get rid of these crawl anomalies and have our pages re-indexed.

  • How is that "obviously" expired? It is just a number in the parameter. It doesn't mean anything to me and it isn't going to mean anything to googlebot. If you use expiring resources in your pages you are going to run into problems with Google rendering the pages. – Stephen Ostermiller Aug 29 at 18:23
  • All video streaming services expire their video links after a while. Our fresh pages do not show md5 and expires parameters to googlebot, and neither does our sitemap, yet google continues to request these really old files. Just like with rnd.js Stephen. – Jason McNoles Aug 29 at 18:36
  • Does this file sent correct MIME type? – Evgeniy Aug 30 at 12:02
  • Yes, it is 'video/mp4'. Even with 'md5' and 'expires' parameters it still returns a 'video/mp4'. The problem is not the mime type, but the fact that google is crawling url resources that haven't existed on the page for over 3 months. We stopped exposing 'md5' and 'expires' parameters, therefore google should crawl only www.example.com/video/12345.mp4 urls now, and not any with md5&expires. – Jason McNoles Aug 30 at 19:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.