Recently, I have enabled all my servers to serve everything over HTTP and HTTPS. Users can access any site via http://www.example.com or https://www.example.com. All pages are identical between the versions, so http://www.example.com/about.php is the same as https://www.example.com/about.php and so on.
Now about that exception. It is in robots.txt:
Apparently this URL must be absolute.
Now the problem I see if that when Google reads https://www.example.com/robots.txt it gets an HTTP sitemap! The documentation on robots.org says that one can specify multiple sitemaps but if I am not sure that putting both the HTTP and HTTPS sitemap is a good idea since they will contain each a list of identical pages (one with HTTP and one with HTTPS).
How should Sitemap in robots.txt be handled for websites that accept HTTP and HTTPS?
Some ideas that came to mind:
- Specify both sitemaps (as mentioned above). Afraid this would cause duplicate content issues.
- Only specify the HTTPS Sitemap. That gives access to all unique pages anyway.
- Find a magical (Apache) way to sent a different robots.txt via HTTP and HTTPS. Is that even possible? Could it cause issues?