2

Previously we've been using Amazon CloudFront for our static content (js, css etc). But to be able to reduce load on our origin servers and to be able to give our international users a good user experience we decided to deliver a couple of our sites through CloudFront.

We noticed very nice drops in page load time, but when checking Google webmaster tools we noticed that all CloudFront-activated sites got a huge drop in pages crawled per day (from avg ~3500 to ~150). Also one of the sites have issues with the Google sitemaps (just marked as "Pending" in GWT) and no new pages or updated pages seems to be updated in the Google SERP. The rest of the sites gets some updates on the Google SERP, but very few compared to before CloudFront activation.

Is there anybody here who have experience in full site delivery through CloudFront (or other CDNs) and effects on SEO/Google? Would be very glad for any insights or suggestions. The risk is that we need to remove CloudFront if this just continues.

  • 1
    You are in effect changing networks, but Google does not know a CDN from anything else. Just changing a network can cause Google to pause a bit sometimes. Google just makes HTTP requests so do a few Fetch as Google(s) to make sure that Google sees things correctly. If after a few pages things look okay, then I would stay the course and be patient. I am leaving room for someone with CDN or CloudFlare experience to answer. – closetnoc Dec 12 '14 at 14:59
  • Thanks! We've been thinking the same, we haven't seen any thing that should be problematic for googlebot in the responses, but we'll do some more checks. We now have cache headers for our dynamic content, otherwise no change. The problem is that the site managers are starting to get nervous because it's been like this since the launch (almost a month ago) – SimonG Dec 12 '14 at 15:24
  • A month can be a bit worry some. I know that sometimes Google takes a bit of a break on my site and slows way down for a period of a few weeks and only picks new pages out of the sitemap. Then all of a sudden, it goes back to about 50k+ pages per day again. I can understand that people are getting worried. But you cannot push Google along and hurry them up. Perhaps someone has ideas for things to check JIC. Keep looking at Google Webmaster Tools for errors. That can be an indication. – closetnoc Dec 12 '14 at 15:33
1

When you added CDN, did you also added site-wide SSL? If you did, your current site in WMT is HTTP and that won't be crawled anymore as you have move away from HTTP.

So go ahead, add a new site in WMT as https:// in front of it and it will show you all the impressions and everything back to normal.

  • Thanks for your answer, but we didn't activate https, it's still http – SimonG Dec 15 '14 at 7:37
0

I find it plausible that the addition of cache headers would reduce the amount of crawling that Googlebot has to do. If you are telling Googlebot that the page can be cached for a week, then it may not come back to crawl again until that cache expires.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.