# Is there a way to disallow crawling of only HTTPS in robots.txt?

I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https only?

According to Wikipedia, each protocol has its own robots.txt

And according to Google's Robots.txt Specification, the robots.txt applies to http AND https

I don't want to Disallow: / for Bing totally, just over https.

• Is there any specific reason to serve both the site both in http and https? I would serve only https if it is available, and make permanent redirects from http URLs to https URLs. – Tero Kilkanen May 29 '14 at 16:16
• @TeroKilkanen Is "hasn't crossed our minds" a specific reason? I'm going to put some thought into this, thanks! – David Wilkins May 29 '14 at 16:58

There is no way to do it in robots.txt itself as served over HTTP.

You could serve a different robots file entirely for secure HTTPS connections. Here is one of doing so using rewrite rules in your .htaccess file:

RewriteEngine On
RewriteCond %{HTTPS} =on
RewriteRule ^robots.txt$robots-deny-all.txt [L]  Where robots-deny-all.txt has the contents: User-agent: * Disallow: /  • I'd avoid doing this, though. It may confuse the crawler into thinking that it should not crawl anything on your site, https or not. – josh3736 May 29 '14 at 19:37 • No it wont Josh, this is fine. When you access robots.txt via https you get the different robots.txt as if you access it over http, as do web crawlers. I have used this method on many website. – Max May 29 '14 at 23:26 • @Chris: It is documented in Google's robots.txt spec. Under "Examples of valid robots.txt URLs:", you can see http://example.com/robots.txt is valid for http://example.com/, but not for https://example.com/. "It is not valid for other subdomains, protocols or port numbers." – Jan Fabry May 30 '14 at 7:53 • Additionally, you can just serve a separate robots.txt for HTTPS requests like this. – zigojacko May 30 '14 at 9:53 Before you try to manipulate robots.txt, ensure that you have defined canonical link elements on your pages. Web crawlers should treat: <link rel="canonical" href="…" />  as a very strong hint that two pages should be considered to have the same content, and that one of the URLs is the preferred address for the content. As stated in RFC 6596 Section 3: The target (canonical) IRI MAY: • Have different scheme names, such as "http" to "https"… With the canonical link hints, a reasonably intelligent web crawler should be able to avoid crawling the site a second time over HTTPS. • Exactly this, and do note it's best to make the https canonical then. True, there is a very small overhead, but that seems hardly something to be worried about. – David Mulder May 29 '14 at 22:30 • However, if you have a canonical link with http on a secure https page, wont there be a warning about an insecure element on the page? Might want to bare this in mind. – Max May 29 '14 at 23:19 • @Max Web browsers don't normally follow the canonical link. – 200_success May 29 '14 at 23:21 • But wont they still see the non-secure element in the page, then display the warning about some elements being insecure. The same as if you had images, js, etc on http? – Max May 29 '14 at 23:23 • I will have to try this, since there is some debate as to whether this would affect user's experiences. – David Wilkins May 30 '14 at 12:16 Add a .htaccess file to redirect HTTPS to HTTP, and to redirect requests for the robots.txt file to one that dissalows HTTPS crawling: # Redirect HTTPS to HTTP RewriteCond %{HTTP:X-Forwarded-Proto} =https RewriteRule ^(.*)$ http://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

RewriteCond %{SERVER_PORT} ^443$RewriteRule ^robots.txt$ robots_https.txt


Then add a robots_https.txt with this in it:

User-agent: *
Disallow: /


Create a separate robots.txt for HTTPS requests, for example: robots_https.txt and place this in the root of your website.

Then add the following lines to your root .htaccess file to redirect all bot requests over HTTPS to use robots_https.txt instead.

RewriteCond %{SERVER_PORT} ^443$RewriteRule ^robots.txt$ robots_https.txt

• Is there any significant difference between this method and the answer that has RewriteCond %{HTTPS} =on ? – David Wilkins May 30 '14 at 12:14
• Same idea. @StephenOstermiller's technique is slightly better, in my opinion, as it avoids magic numbers. – 200_success May 30 '14 at 12:19
• No not really, one is based on HTTPS protocol and the other is based on server port for HTTPS. Both will work well in this instance. I'd probably be inclined to opt for StephenOstermiller's version as well. – zigojacko May 30 '14 at 12:20