My website (
www.example.com) has all of my images hosted via CDN under a different sub-domain (
We've noticed an odd behavior recently, that seems as though
Googlebot-Image/1.0 is trying to crawl valid image URLs from the CDN domain but prefixed with our site domain (or adding a leading slash?).
e.g. instead of requesting the image at
https://cdn.example.com/product/12345.jpg, it is attempting to request
This has resulted in hundreds of thousands of exceptions per week being logged on our main site, as the actual URLs are invalid.
Is there anything I can do to mitigate this, other than programming the site itself to handle the malformed URLs?
Would disallowing Googlebot-Image on the
www domain prevent it from successfully crawling the
Do I need a robots.txt file on the
cdn domain itself?