2

Recently, I have enabled all my servers to serve everything over HTTP and HTTPS. Users can access any site via http://www.example.com or https://www.example.com. All pages are identical between the versions, so http://www.example.com/about.php is the same as https://www.example.com/about.php and so on.

URLs are relative, so they do not mention the protocol with one exception. In other words, if the page is loaded with HTTP, it will link to other pages, images, CSS, Javascript over HTTP and the same with HTTPS, as to avoid mixed content warnings.

Now about that exception. It is in robots.txt:

Sitemap: http://www.example.com/sitemap.php

Apparently this URL must be absolute.

Now the problem I see if that when Google reads https://www.example.com/robots.txt it gets an HTTP sitemap! The documentation on robots.org says that one can specify multiple sitemaps but if I am not sure that putting both the HTTP and HTTPS sitemap is a good idea since they will contain each a list of identical pages (one with HTTP and one with HTTPS).

How should Sitemap in robots.txt be handled for websites that accept HTTP and HTTPS?

Some ideas that came to mind:

  • Specify both sitemaps (as mentioned above). Afraid this would cause duplicate content issues.
  • Only specify the HTTPS Sitemap. That gives access to all unique pages anyway.
  • Find a magical (Apache) way to sent a different robots.txt via HTTP and HTTPS. Is that even possible? Could it cause issues?
  • Google uses sitemaps as one way to determine which of your URLs are canonical. So put the version you want Google to send traffic to in your sitemap. See The Sitemap Paraox – Stephen Ostermiller Jun 27 '17 at 18:21
  • Any particular reason why you're not explicitly setting HTTPS as canonical and redirecting? – DocRoot Jun 27 '17 at 18:29
  • @DocRoot - That brings up a few questions! I've been using canonicals for a long time (on sites that need them) but is there ever a reason to use canonicals and redirect? – Itai Jun 27 '17 at 19:46
  • Canonical link tag is strong hint for search engine to which page you want to index into search result. If you think you can index both version for same page then that's not true. When you provide both version then Google will crawl both pages, it means waste of bandwidth and Google may delay to index your other important pages as well. Google bot comes to your site with limited crawl budget. Read 3rd FAQ. – Goyllo Jun 28 '17 at 7:54
0

A sitemap at http://www.example.com/sitemap.php can only contain URLs from http://www.example.com/.¹ The scheme and the host must be the same.

So if you 1) want to provide sitemaps for both protocols, and 2) link both sitemaps via the Sitemap field in the robots.txt, you have to provide separate robots.txt files for HTTP and HTTPS:

#        http://www.example.com/robots.txt

Sitemap: http://www.example.com/sitemap.php
#        https://www.example.com/robots.txt

Sitemap: https://www.example.com/sitemap.php

(It should be easy to achieve this with Apache, see for example the answers to Is there a way to disallow crawling of only HTTPS in robots.txt?)

But you might want to provide a sitemap only for the canonical variant (e.g., only for HTTPS), because there is not much point in letting search engines parse the sitemap for the non-canonical variant, as they typically wouldn’t want to index any of its URLs. So if HTTPS should be canonical:

  1. On each HTTP page, link to its HTTPS version with the canonical link type.
  2. Provide a sitemap only on HTTPS, listing only the HTTPS URLs.
  3. Link the sitemap (ideally only) from the HTTPS robots.txt.

¹ Except if cross submits are used.

  • Excellent and very comprehensive. – Itai Jun 28 '17 at 17:07
0
http://www.example.com/about/
http://www.example.com/about
http://example.com/about/
http://example.com/about
https://www.example.com/about/
https://www.example.com/about

These kind of duplicate content Google already handling from many years ago. So first of don't worry about duplicate content issue.

It is totally fine to serve HTTP and HTTPS version of site on same time, specially when you're migrating your site from HTTP to HTTPS, Stackoverflow also done that in past.

Here Google will index only one version of your webpage, it means they will not going to index both version http://www.example.com/about.php and https://www.example.com/about.php. In most of time, by default they will choose HTTPS

And again there is no need to add your sitemap file into robots.txt. Specially when you think about Google(It is not ask.com), because they gives us option to submit our sitemap into webmaster tool. So create two properties into search console like http://www.example.com and https://www.example.com and submit individual sitemap there.

I don't know why you're so serious about sitemap, robots.txt and all thing. Google can crawl and index any website without sitemap, for example wikipedia does not have any sitemap, but it is crawl often, because they have good internal link structure.

  • Google recommends submitting sitemaps and the robots.txt file serves useful purposes, so I don't want to remove them unless there is a compelling reason to. – Itai Jun 27 '17 at 18:20
  • Google never recommends that. It says we can capture your sitemap file from robots.txt as well, but if you already submitted into search console then it's totally fine. If your sitemap file is public then anybody can scrape your site data. If your main concern is Google, then you're overthinking right now. – Goyllo Jun 27 '17 at 18:24
  • Great link. One of the clearest text Google published! OK. I did submit the HTTPS sitemap now since the other one was already there. I'll wait a week or so to see what happens. – Itai Jun 27 '17 at 19:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.