Timeline for Is there a way to disallow crawling of only HTTPS in robots.txt?
Current License: CC BY-SA 3.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 3, 2014 at 12:00 | vote | accept | David Wilkins | ||
| May 30, 2014 at 9:53 | comment | added | zigojacko |
Additionally, you can just serve a separate robots.txt for HTTPS requests like this.
|
|
| May 30, 2014 at 7:53 | comment | added | Jan Fabry |
@Chris: It is documented in Google's robots.txt spec. Under "Examples of valid robots.txt URLs:", you can see http://example.com/robots.txt is valid for http://example.com/, but not for https://example.com/. "It is not valid for other subdomains, protocols or port numbers."
|
|
| May 29, 2014 at 23:26 | comment | added | Max | No it wont Josh, this is fine. When you access robots.txt via https you get the different robots.txt as if you access it over http, as do web crawlers. I have used this method on many website. | |
| May 29, 2014 at 19:37 | comment | added | josh3736 | I'd avoid doing this, though. It may confuse the crawler into thinking that it should not crawl anything on your site, https or not. | |
| May 29, 2014 at 18:26 | history | edited | Stephen Ostermiller♦ | CC BY-SA 3.0 |
clarity as suggested by w3d
|
| May 29, 2014 at 14:39 | history | edited | Stephen Ostermiller♦ | CC BY-SA 3.0 |
added 4 characters in body
|
| May 29, 2014 at 14:25 | history | answered | Stephen Ostermiller♦ | CC BY-SA 3.0 |