My website (www.example.com) has all of my images hosted via CDN under a different sub-domain (cdn.example.com).

We've noticed an odd behavior recently, that seems as though Googlebot-Image/1.0 is trying to crawl valid image URLs from the CDN domain but prefixed with our site domain (or adding a leading slash?).

e.g. instead of requesting the image at https://cdn.example.com/product/12345.jpg, it is attempting to request https://www.example.com/https:/cdn.example.com/product/12345.jpg

This has resulted in hundreds of thousands of exceptions per week being logged on our main site, as the actual URLs are invalid.

Is there anything I can do to mitigate this, other than programming the site itself to handle the malformed URLs?

Would disallowing Googlebot-Image on the www domain prevent it from successfully crawling the cdn domain?

Do I need a robots.txt file on the cdn domain itself?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.