I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https only?

According to Wikipedia, each protocol has its own robots.txt

And according to Google's Robots.txt Specification, the robots.txt applies to http AND https

I don't want to Disallow: / for Bing totally, just over https.

  • 5
    Is there any specific reason to serve both the site both in http and https? I would serve only https if it is available, and make permanent redirects from http URLs to https URLs. May 29, 2014 at 16:16
  • @TeroKilkanen Is "hasn't crossed our minds" a specific reason? I'm going to put some thought into this, thanks! May 29, 2014 at 16:58

4 Answers 4


There is no way to do it in robots.txt itself as served over HTTP.

You could serve a different robots file entirely for secure HTTPS connections. Here is one of doing so using rewrite rules in your .htaccess file:

RewriteEngine On
RewriteCond %{HTTPS} =on
RewriteRule ^robots.txt$ robots-deny-all.txt [L]

Where robots-deny-all.txt has the contents:

User-agent: *
Disallow: /
  • 3
    I'd avoid doing this, though. It may confuse the crawler into thinking that it should not crawl anything on your site, https or not.
    – josh3736
    May 29, 2014 at 19:37
  • 1
    No it wont Josh, this is fine. When you access robots.txt via https you get the different robots.txt as if you access it over http, as do web crawlers. I have used this method on many website.
    – Max
    May 29, 2014 at 23:26
  • 2
    @Chris: It is documented in Google's robots.txt spec. Under "Examples of valid robots.txt URLs:", you can see http://example.com/robots.txt is valid for http://example.com/, but not for https://example.com/. "It is not valid for other subdomains, protocols or port numbers."
    – Jan Fabry
    May 30, 2014 at 7:53
  • Additionally, you can just serve a separate robots.txt for HTTPS requests like this.
    – zigojacko
    May 30, 2014 at 9:53

Before you try to manipulate robots.txt, ensure that you have defined canonical link elements on your pages.

Web crawlers should treat:

<link rel="canonical" href="…" />

as a very strong hint that two pages should be considered to have the same content, and that one of the URLs is the preferred address for the content.

As stated in RFC 6596 Section 3:

The target (canonical) IRI MAY:

  • Have different scheme names, such as "http" to "https"…

With the canonical link hints, a reasonably intelligent web crawler should be able to avoid crawling the site a second time over HTTPS.

  • Exactly this, and do note it's best to make the https canonical then. True, there is a very small overhead, but that seems hardly something to be worried about. May 29, 2014 at 22:30
  • However, if you have a canonical link with http on a secure https page, wont there be a warning about an insecure element on the page? Might want to bare this in mind.
    – Max
    May 29, 2014 at 23:19
  • @Max Web browsers don't normally follow the canonical link. May 29, 2014 at 23:21
  • But wont they still see the non-secure element in the page, then display the warning about some elements being insecure. The same as if you had images, js, etc on http?
    – Max
    May 29, 2014 at 23:23
  • I will have to try this, since there is some debate as to whether this would affect user's experiences. May 30, 2014 at 12:16

Add a .htaccess file to redirect HTTPS to HTTP, and to redirect requests for the robots.txt file to one that dissalows HTTPS crawling:

# Redirect HTTPS to HTTP
RewriteCond %{HTTP:X-Forwarded-Proto} =https
RewriteRule ^(.*)$ http://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

RewriteCond %{SERVER_PORT} ^443$
RewriteRule ^robots.txt$ robots_https.txt

Then add a robots_https.txt with this in it:

User-agent: *
Disallow: /

Create a separate robots.txt for HTTPS requests, for example: robots_https.txt and place this in the root of your website.

Then add the following lines to your root .htaccess file to redirect all bot requests over HTTPS to use robots_https.txt instead.

RewriteCond %{SERVER_PORT} ^443$ 
RewriteRule ^robots.txt$ robots_https.txt
  • Is there any significant difference between this method and the answer that has RewriteCond %{HTTPS} =on ? May 30, 2014 at 12:14
  • Same idea. @StephenOstermiller's technique is slightly better, in my opinion, as it avoids magic numbers. May 30, 2014 at 12:19
  • No not really, one is based on HTTPS protocol and the other is based on server port for HTTPS. Both will work well in this instance. I'd probably be inclined to opt for StephenOstermiller's version as well.
    – zigojacko
    May 30, 2014 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.