Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https only?

According to Wikipedia, each protocol has its own robots.txt

And according to Google's Robots.txt Specification, the robots.txt applies to http AND https

I don't want to Disallow: / for Bing totally, just over https.

share|improve this question
5  
Is there any specific reason to serve both the site both in http and https? I would serve only https if it is available, and make permanent redirects from http URLs to https URLs. –  Tero Kilkanen May 29 at 16:16
    
@TeroKilkanen Is "hasn't crossed our minds" a specific reason? I'm going to put some thought into this, thanks! –  David Wilkins May 29 at 16:58

3 Answers 3

up vote 8 down vote accepted

There is no way to do it in robots.txt itself as served over HTTP.

You could serve a different robots file entirely for secure HTTPS connections. Here is one of doing so using rewrite rules in your .htaccess file:

RewriteEngine On
RewriteCond %{HTTPS} =on
RewriteRule ^robots.txt$ robots-deny-all.txt [L]

Where robots-deny-all.txt has the contents:

User-agent: *
Disallow: /
share|improve this answer
3  
I'd avoid doing this, though. It may confuse the crawler into thinking that it should not crawl anything on your site, https or not. –  josh3736 May 29 at 19:37
1  
No it wont Josh, this is fine. When you access robots.txt via https you get the different robots.txt as if you access it over http, as do web crawlers. I have used this method on many website. –  Max May 29 at 23:26
1  
@Chris: It is documented in Google's robots.txt spec. Under "Examples of valid robots.txt URLs:", you can see http://example.com/robots.txt is valid for http://example.com/, but not for https://example.com/. "It is not valid for other subdomains, protocols or port numbers." –  Jan Fabry May 30 at 7:53
    
Additionally, you can just serve a separate robots.txt for HTTPS requests like this. –  zigojacko May 30 at 9:53

Before you try to manipulate robots.txt, ensure that you have defined canonical link elements on your pages.

Web crawlers should treat:

<link rel="canonical" href="…" />

as a very strong hint that two pages should be considered to have the same content, and that one of the URLs is the preferred address for the content.

As stated in RFC 6596 Section 3:

The target (canonical) IRI MAY:

  • Have different scheme names, such as "http" to "https"…

With the canonical link hints, a reasonably intelligent web crawler should be able to avoid crawling the site a second time over HTTPS.

share|improve this answer
    
Exactly this, and do note it's best to make the https canonical then. True, there is a very small overhead, but that seems hardly something to be worried about. –  David Mulder May 29 at 22:30
    
However, if you have a canonical link with http on a secure https page, wont there be a warning about an insecure element on the page? Might want to bare this in mind. –  Max May 29 at 23:19
    
@Max Web browsers don't normally follow the canonical link. –  200_success May 29 at 23:21
    
But wont they still see the non-secure element in the page, then display the warning about some elements being insecure. The same as if you had images, js, etc on http? –  Max May 29 at 23:23
    
I will have to try this, since there is some debate as to whether this would affect user's experiences. –  David Wilkins May 30 at 12:16

Create a separate robots.txt for HTTPS requests, for example: robots_https.txt and place this in the root of your website.

Then add the following lines to your root .htaccess file to redirect all bot requests over HTTPS to use robots_https.txt instead.

RewriteCond %{SERVER_PORT} ^443$ 
RewriteRule ^robots.txt$ robots_https.txt
share|improve this answer
    
Is there any significant difference between this method and the answer that has RewriteCond %{HTTPS} =on ? –  David Wilkins May 30 at 12:14
    
Same idea. @StephenOstermiller's technique is slightly better, in my opinion, as it avoids magic numbers. –  200_success May 30 at 12:19
    
No not really, one is based on HTTPS protocol and the other is based on server port for HTTPS. Both will work well in this instance. I'd probably be inclined to opt for StephenOstermiller's version as well. –  zigojacko May 30 at 12:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.