3

I am migrating a website from http to https entirely, all http urls will have 301 redirects to their https counterparts.

From https://support.google.com/webmasters/answer/6033049

We reference our HTTP sitemaps in robots.txt. Should we update the robots.txt to include our new HTTPS sitemaps?

We recommend separate robots.txt files for HTTP and HTTPS, pointing to separate sitemap files for HTTP and HTTPS. We also recommend listing a specific URL in only one sitemap file.

What URLs should our sitemaps list if we have redirects (from HTTP to HTTPS or the reverse)?

List all HTTP URLs in your HTTP sitemap, and all HTTPS URLs in your HTTPS sitemap, regardless of redirects when the user visits the page. Having pages listed in your sitemap regardless of redirects will help search engines discover the new URLs faster.

From this I assume the following should be correct:

  1. http://example.com/robots.txt should exist and have a Sitemap directive pointing to the old sitemap.xml with http urls.

  2. https://example.com/robots.txt should exist and have a Sitemap directive pointing to the new sitemap.xml (maybe called something like sitemap_https.xml) with https urls that are same as the old ones but have https instead of http.

But further reading of google guidelines shows another approach that contradicts this one (or maybe I just misunderstood something?)

From answer https://support.google.com/webmasters/answer/6033080:

Update your robots.txt files:

  • On the source site, remove all robots.txt directives. This allows Googlebot to discover all redirects to the new site and update our index.

  • On the destination site, make sure the robots.txt file allows all crawling. This includes crawling of images, CSS, JavaScript, and other page assets, apart from the URLs you are certain you do not want crawled.

On the destination site, submit the two sitemaps you prepared previously containing the old and new URLs. This helps our crawlers discover the redirects from the old URLs to the new URLs, and facilitates the site move.

This is how I understand this approach:

  1. http robots.txt should exist and have no directives in it (be empty).

  2. https robots.txt should exist and have two Sitemap directives, one to old sitemap.xml and another to new sitemap_https.xml

Maybe "submit the two sitemaps" means something different from listing them in robots.txt? Like using the Search Console or something? It doesn't clarify, just "submit"...

Besides, point 1 of this approach contradicts point 1 of the first approach.

  • Well... that is a whole lotta stanky pooh. Leave it up to Google to overly complicate things with their brand of hooey. What you were reading assumes that both HTTP and HTTPS would be available. In your case, you already redirect HTTP to HTTPS. You are done. Just make sure your sitemap has HTTPS in its URLs. Done. Go drink a beer and stop reading so much. It will make you go cross-eyed! [insert friendly grin] Cheers!! – closetnoc Mar 8 '17 at 23:28
  • It is about not losing rankings while moving to HTTPS. And for that reason google says I have to provide two sitemaps - one with old HTTP URLs and one with new HTTPS counterparts, regardless of redirects. Otherwise their bots will need more time to discover the redirects (yeah, it seems strange but that's what google says and google is always right... isn't he?) – Vilial Mar 9 '17 at 21:28
  • Google is a socialist! ;-0 Oh Snap! Did he really say that?? People simply do a blanket redirect (redirect all requests) from HTTP to HTTPS and Google discovers things just fine - really. Would I lie to you?? Except about fishing I mean. The truth of the matter is if a blanket redirect is done, then Google does not need anything else. It seems like Google lives in a bubble. – closetnoc Mar 9 '17 at 23:24
  • @Vilial, hey. I've faced with the same problem just now. Could you please tell me how did you resolve this issue? What did you decide to use: the first approach or the second one? Thanks a lot! – Vladyslav Turak Sep 15 '17 at 20:40
  • 1
    I'm doing separate robots.txt files since redirecting robots.txt is generally not very good according to many sources. I'm putting old sitemap to old robots file and new to new one. And submittimg both sitemaps for the new domain in the search console. Dunno if it's the best practice though, there's a lot of conflicting information on this question – Vilial Sep 16 '17 at 6:26
4

The first approach is the correct one. We successfully migrated a high traffic and rankings website from HTTP to HTTPS completely. The approach based on Google guidelines was:

  1. All HTTP URLs do a 301 permanent redirect to HTTPS.

  2. http://www.example.com/robots.txt would redirect to the HTTPS version https://www.example.com/robots.txt

  3. The new sitemap shall have all HTTPS links.

Here is a good post about this from Google :

https://plus.google.com/+JohnMueller/posts/PY1xCWbeDVC

2

If you are keeping both HTTP and HTTPS and are not planning to redirect everything to HTTPS, then maybe Google's advice makes sense. But besides that it seems like strange advice to me.

Presumably you want to move everything to HTTPS eventually, so you should use HTTPS URLs wherever possible. Your robots.txt file would show your HTTPS sitemap link on both http://example.com/robots.txt and https://example.com/robots.txt. And similarly for the sitemap, it would show HTTPS URLs on both versions.

This is much easier from a technical perspective, and will prioritise HTTPS URLs in Google.

  • If HTTP is redirecting, wouldn't http://example.com/robots.txt redirect to https://example.com/robots.txt? I'd think that would be a fine approach. – Stephen Ostermiller Apr 13 '17 at 9:32
  • @Stephen In my answer I'm talking about not redirecting. If you're moving to HTTPS the first stage should be to set up all canonical links to be HTTPS, give Google time to find the links (and yourself to check everything works fine on HTTPS) before doing the 301s. So robots.txt wouldn't redirect at this point, but it can point to your HTTPS sitemap. – DisgruntledGoat Apr 13 '17 at 10:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.