5

I have somewhere close to 500,000 user-uploaded images hosted on a Cloudfront CDN -- separate from our main host (exampledomain.com). Up until this point, few of them have been getting indexed at the default distribution URLs. Example:

https://d7oxxxxxxx.cloudfront.net/images/example_directory/subdirectory/LG_example_filename.jpg

So I added a CNAME (alternate domain name) so that the URLs have now become: http://media.exampledomain.com/images/example_directory/subdirectory/LG_example_filename.jpg

And I added "media.exampledomain.com" as a verified domain in Google Search Console.

I also have a dynamic sitemap hosted on exampledomain.com that lists all of the images I would want to get indexed -- one image per page (there are probably close to 240,000 pages altogether). Example:

<url>
<loc>http://www.exampledomain.com/directory/pagename</loc>
<changefreq>daily</changefreq>
<image:image>
<image:loc>
http://media.exampledomain.com/images/exampledirectory/subdirectory/LG_filname.jpg
</image:loc>
<image:title>Example Image Title</image:title>
<image:caption>Example Image Caption</image:caption>
</image:image>
</url>

According to what I've read, this should get Google to start indexing all of the images. However, I do not want to potentially wait a whole week to find out that there is something else I may not have done or that something else may be blocking the images from being indexed. The Cloudfront URLs are all fully public as far as I can tell and there aren't any robots.txt restrictions in place on the CDN. I only have one Cloudfront distribution currently active so I don't believe there should be any issues with duplicate content. Is there anything else I may need to account for or some way I can see in advance if it is going to work?

Thanks for any help you can provide.

UPDATE:

I've been tracking this for a few days now. The Google bots have been crawling and indexing all of our site's pages at a nice swift rate (over 50,000 pages in a day!). However, there is still something up with the images. I see that there are over 160,000 images submitted in the sitemap and Google has crawled roughly 15,000 of them, but only 50 have actually been indexed. Does anyone have any ideas why Google may be having difficulty with these?

Here is an example format for one of URLs. There is a 12-14 digit timestamp appended to the end of all of the files:

http://media.exampledomain.com/images/category/id/LG_keywords_1442182082.5437.jpg

Improve this question
2
  • I did a URL change for all my images on 9/4/15, with proper 301 redirect. The indexed count dropped from around 150 to 0, today it's at 27. Slowly increasing day by day. I think indexing images is a bit slower than pages. Maybe you just need some patience? :)
    – Thomas Jensen
    Commented Sep 14, 2015 at 10:34
  • Maybe you are right. I'm seeing that 156 have now been indexed. There are also some crawl errors reported, where it's finding some incorrect or missing links -- a few of them having %7c%7c%7c tacked on to the end of the link -- which is mysterious to me, and I suspect that is keeping the crawl rate down a little.
    – Ian Spangler
    Commented Sep 14, 2015 at 23:48

1 Answer 1

Reset to default
2

That is pretty much what I have done;

  • Images on CDN with CNAME record.
  • Verified the CDN domain in Google Webmasters.
  • Using CDN urls in sitemap.
  • Added sitemap listing in robots.txt as well as in Google Webmasters.
  • No robots restriction on CDN domain.

And Google is indexing my images just fine. If I search for site:mysitedomain.com on Google; all images from the CDN is shown as well :)

Update:

In the robots.txt file for my website I have:

User-agent: *
Disallow: /harming/humans
Sitemap: http://www.website.net/sitemap.xml

This makes sure that other search engines (not Google) also finds the sitemap. More info here: http://www.sitemaps.org/protocol.html#submit_robots

The robots.txt on my CDN domain simply allows crawling and looks like this:

User-agent: *
Disallow:
Improve this answer
3
  • Thanks. Could you elaborate on point number 4 a bit? Are you simply referencing sitemap: /exampledomain.com/sitemap.xml in the main host's robots.txt? And is there anything else that needs to be done with the CDN (media.exampledomain.com) in Webmasters console or anything that should be done in the CDN's robots.txt file too?
    – Ian Spangler
    Commented Sep 9, 2015 at 22:16
  • Updated answer. All I have done with the CDN domain in Webmasters is validate that I am the owner of that domain.
    – Thomas Jensen
    Commented Sep 9, 2015 at 22:25
  • Great, that is pretty much what I have going. I'll let you know how things work out!
    – Ian Spangler
    Commented Sep 9, 2015 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.