7

I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https only?

According to Wikipedia, each protocol has its own robots.txt

And according to Google's Robots.txt Specification, the robots.txt applies to http AND https

I don't want to Disallow: / for Bing totally, just over https.

Improve this question
2
  • 5
    Is there any specific reason to serve both the site both in http and https? I would serve only https if it is available, and make permanent redirects from http URLs to https URLs.
    – Tero Kilkanen
    Commented May 29, 2014 at 16:16
  • @TeroKilkanen Is "hasn't crossed our minds" a specific reason? I'm going to put some thought into this, thanks!
    – David Wilkins
    Commented May 29, 2014 at 16:58

4 Answers 4

Reset to default
13

There is no way to do it in robots.txt itself as served over HTTP.

You could serve a different robots file entirely for secure HTTPS connections. Here is one of doing so using rewrite rules in your .htaccess file:

RewriteEngine On
RewriteCond %{HTTPS} =on
RewriteRule ^robots.txt$ robots-deny-all.txt [L]

Where robots-deny-all.txt has the contents:

User-agent: *
Disallow: /
Improve this answer
4
  • 3
    I'd avoid doing this, though. It may confuse the crawler into thinking that it should not crawl anything on your site, https or not.
    – josh3736
    Commented May 29, 2014 at 19:37
  • 1
    No it wont Josh, this is fine. When you access robots.txt via https you get the different robots.txt as if you access it over http, as do web crawlers. I have used this method on many website.
    – Max
    Commented May 29, 2014 at 23:26
  • 2
    @Chris: It is documented in Google's robots.txt spec. Under "Examples of valid robots.txt URLs:", you can see http://example.com/robots.txt is valid for http://example.com/, but not for https://example.com/. "It is not valid for other subdomains, protocols or port numbers."
    – Jan Fabry
    Commented May 30, 2014 at 7:53
  • Additionally, you can just serve a separate robots.txt for HTTPS requests like this.
    – zigojacko
    Commented May 30, 2014 at 9:53
4

Before you try to manipulate robots.txt, ensure that you have defined canonical link elements on your pages.

Web crawlers should treat:

<link rel="canonical" href="…" />

as a very strong hint that two pages should be considered to have the same content, and that one of the URLs is the preferred address for the content.

As stated in RFC 6596 Section 3:

The target (canonical) IRI MAY:

…

  • Have different scheme names, such as "http" to "https"…

With the canonical link hints, a reasonably intelligent web crawler should be able to avoid crawling the site a second time over HTTPS.

Improve this answer
5
  • Exactly this, and do note it's best to make the https canonical then. True, there is a very small overhead, but that seems hardly something to be worried about.
    – David Mulder
    Commented May 29, 2014 at 22:30
  • However, if you have a canonical link with http on a secure https page, wont there be a warning about an insecure element on the page? Might want to bare this in mind.
    – Max
    Commented May 29, 2014 at 23:19
  • @Max Web browsers don't normally follow the canonical link.
    – 200_success
    Commented May 29, 2014 at 23:21
  • But wont they still see the non-secure element in the page, then display the warning about some elements being insecure. The same as if you had images, js, etc on http?
    – Max
    Commented May 29, 2014 at 23:23
  • I will have to try this, since there is some debate as to whether this would affect user's experiences.
    – David Wilkins
    Commented May 30, 2014 at 12:16
1

Add a .htaccess file to redirect HTTPS to HTTP, and to redirect requests for the robots.txt file to one that dissalows HTTPS crawling:

# Redirect HTTPS to HTTP
RewriteCond %{HTTP:X-Forwarded-Proto} =https
RewriteRule ^(.*)$ http://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

RewriteCond %{SERVER_PORT} ^443$
RewriteRule ^robots.txt$ robots_https.txt

Then add a robots_https.txt with this in it:

User-agent: *
Disallow: /
Improve this answer
0

Create a separate robots.txt for HTTPS requests, for example: robots_https.txt and place this in the root of your website.

Then add the following lines to your root .htaccess file to redirect all bot requests over HTTPS to use robots_https.txt instead.

RewriteCond %{SERVER_PORT} ^443$ 
RewriteRule ^robots.txt$ robots_https.txt
Improve this answer
3
  • Is there any significant difference between this method and the answer that has RewriteCond %{HTTPS} =on ?
    – David Wilkins
    Commented May 30, 2014 at 12:14
  • Same idea. @StephenOstermiller's technique is slightly better, in my opinion, as it avoids magic numbers.
    – 200_success
    Commented May 30, 2014 at 12:19
  • No not really, one is based on HTTPS protocol and the other is based on server port for HTTPS. Both will work well in this instance. I'd probably be inclined to opt for StephenOstermiller's version as well.
    – zigojacko
    Commented May 30, 2014 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.