We switched to HTTPS in our legacy ASP.NET site about a month ago. We had already implemented redirection with HTTP status code 301 'Moved permanently' for outdated aspx resources in the app's Global.asax file before switching to HTTPS, so we added a few more lines to the redirect algorithm in the Application_BeginRequest method to redirect all requests to aspx pages to their HTTPS counterparts. The core part of the current version of this algorithm looks like this ("example.com" is used instead of the real domain):
Sub Application_BeginRequest(ByVal sender As Object, ByVal e As EventArgs) Dim host_requested As String = Request.Url.AbsoluteUri ' Save the original request string to know whether we need to redirect Dim host_to_redirect As String = host_requested ' Prepare for analysis - remove the protocol and domain For Each sRoot In m_asPossibleRoots If host_to_redirect.StartsWith(sRoot, StringComparison.OrdinalIgnoreCase) Then host_to_redirect = host_to_redirect.Substring(sRoot.Length) Exit For End If Next ' Remove "home/" if any Dim sHome As String = "home/" If host_to_redirect.StartsWith(sHome, StringComparison.OrdinalIgnoreCase) Then host_to_redirect = host_to_redirect.Substring(sHome.Length) End If ' Redirect old product pages and articles If host_to_redirect.StartsWith("outdated-resource-folder1", StringComparison.OrdinalIgnoreCase) Then host_to_redirect = "resource1.aspx" ElseIf host_to_redirect.StartsWith("outdated-resource-folder2", StringComparison.OrdinalIgnoreCase) Then host_to_redirect = "resource2.aspx" ' ElseIf ' ... processing other outdated resources End If ' ... other manipulations with host_to_redirect ' Add the required protocol and domain host_to_redirect = "https://example.com/" + host_to_redirect ' If we need to redirect, do it If (host_to_redirect <> host_requested) Then ' also provides a 301 HTTP status code in the response Response.RedirectPermanent(host_to_redirect) End If End Sub
We also added the HTTPS version of our website to Google Webmaster Tools and similar resources working with our website. All works Ok, but recently it has turned out that one of the web-analytics tool we use determines the main protocol of our resource (HTTP or HTTPS) by sending requests to example.com/robots.txt and reading the protocol in the response. Sure, all these requests to robots.txt on our website are still processed as HTTP because we didn't (and couldn't) code redirection for this non-ASP.NET resource in Global.asax.
Having all this, the main question is the following: Do we need to implement HTTPS redirection for robots.txt if we switched to HTTPS in our ASP.NET website? May it affect SEO and similar things? We didn't bother about this as it seems this is not needed for Google & Bing WMT, Google Analytics and similar tools.
If we need to implement HTTPS redirections for robots.txt though, do we need to do this for other non-ASP.NET resources like sitemap.xml and the like? And what is the best method to implement this in our ASP.NET project? As I understand, we can do this only in web.config, but this may mean that we need to move the HTTPS redirection logic part from Global.asax to web.config too. Will this 2-stage redirection logic work seamlessly?