I am migrating my well indexed website from HTTP to HTTPS only.
Steps planned:
- All HTML pages canonical to be HTTPS now
- All HTTP pages will 301 redirect to HTTPS with the same URL
- All sitemaps will have HTTPS URLs
- All links on HTML pages will be HTTPS
Issue 1:
Currently robots.txt
at HTTPS level, disallow all, as we did not want crawler to crawl HTTPS pages.
What should be our approach now: Allowing https now for all links is obvious.
But what about allowing / disallowing HTTP (as if it did not crawl HTTP how would it know that nothing new has happened, only HTTP has been migrated to HTTPS) Though 301 will indicate that but that will only happen when crawler opens the same page with HTTP.
Issue 2:
If in sitemap I add all HTTPS links, it lands me to same question above. Crawler will start crawling https links and index them. But we want it to know that it is the same HTTP version and pass on rankings
So ideally for some span of time until all our links are indexed with HTTPS should we have both HTTP and HTTPS in sitemaps?
As per current understanding, following is the plan:
If google stops crawling http it won't pass rankings. So i am planning to allow both http and https as per robots are concerned.
But don't know how they will crawl it as on our web-server, we are 301 redirecting all http links to https counterpart. So even if it crawls root of my site say http://www.example.com it will eventually redirect to https://www.example.com where it will find all https links and hence will follow that.
May be it also try to crawl the links it has already indexed(all http links) and will find that they are redirected, but that depends on crawler behavior and frequency. In the time span between crawler's understanding of 301 and https version also crawled it will create duplicate content issue and hence affect our rankings.