3

My website (www.example.com) has all of my images hosted via CDN under a different sub-domain (cdn.example.com).

We've noticed an odd behavior recently, that seems as though Googlebot-Image/1.0 is trying to crawl valid image URLs from the CDN domain but prefixed with our site domain (or adding a leading slash?).

e.g. instead of requesting the image at https://cdn.example.com/product/12345.jpg, it is attempting to request https://www.example.com/https:/cdn.example.com/product/12345.jpg

This has resulted in hundreds of thousands of exceptions per week being logged on our main site, as the actual URLs are invalid.

Is there anything I can do to mitigate this, other than programming the site itself to handle the malformed URLs?

Would disallowing Googlebot-Image on the www domain prevent it from successfully crawling the cdn domain?

Do I need a robots.txt file on the cdn domain itself?

1

This is likely due to a malformed URL in your web pages. Notice that https://www.example.com/https:/cdn.example.com/product/12345.jpg only has one slash in https:/cdn instead of two. If your page has a <img src="https:/cdn.example.com/..." it may be interpreted as a relative link.

The solution would be to find all image URLs in your site and make sure they are well formed with two slashes. Alternately, you could use protocol relative linking that starts with two slashes: <img src="//cdn.example.com/...".

On your web server, you could redirect URLs that contains "http:/" like this:

RewriteEngine On
RewriteCond %{THE_REQUEST} .*(https?):/{1,2}(cdn\.example\.com/.*) [NC]
RewriteRule ^(.*)$ %1://%2 [R=301,L]

This will handle all the following requests having http or https, one or two slashes, and optionally a path prefix:

  • https://www.example.com/https:/cdn.example.com/product/12345.jpg
  • https://www.example.com/http:/cdn.example.com/product/12345.jpg
  • https://www.example.com/https://cdn.example.com/product/12345.jpg
  • https://www.example.com/some-path/https:/cdn.example.com/product/12345.jpg
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.