Can a webpage differ in content if 'http' is changed to 'https' or if 'www.' is added after 'http://' (or 'https://')?

Question

When I use the Python package newspaper3k package and run the code

import newspaper
paper = newspaper.build('http://abcnews.com', memoize_articles=False)
for url in paper.article_urls():
    print(url)

I get a list of URLs for articles that I can download, in which both these URLs exist

• http://abcnews.go.com/Health/coronavirus-transferred-animals-humans-scientists-answer/story?id=73055380
• https://abcnews.go.com/Health/coronavirus-transferred-animals-humans-scientists-answer/story?id=73055380

As can be seen, the only difference between the two URLs is the s in https.

The question is, can the webpage content differ simply because an s is added to http? If I scrape a news source (in this case http://abcnews.com), do I need to download both articles to be sure I don't miss any article, or are they guaranteed to have the same content so that I can download only one of them?

I have also noticed that some URLs also are duplicated by adding www. after the http:// (or https://). I have the same question here: Can this small change cause the webpage content to differ, and is this something I should take into account or can I simply ignore one of these two URLs?

Answer 1

It is possible to display different content for http or https by redirecting via htaccess, but then it would be a different page that would be loading.

https is simply loading a SSL certificate, I suspect in very close to 100% of cases, the same page with the same content would be loaded.

Answer 2

Steve's answer is mostly correct: it is indeed possible for the content to differ between http and https versions of the page.

One correction I'd suggest is that you don't actually need to do a redirect to accomplish this. For instance, with the Apache webserver, you could use the "Listen" directive to direct requests to different DocumentRoots. (Other webservers, such as Nginx have similar capabilities.) Alternatively, a server side language might also detect whether the connection is secure and display different content accordingly.

Though somewhat technical, the reason this distinction is important is that in a redirect, you'll have a different URL (e.g., http://example.com/unsecure redirects to https://example.com/secure). But if done without a redirect you absolutely can have different content when the URLs are exactly the same except for the protocol, i.e., whether the connection is http or https. (Thus, http://example.com/page could be different from https://example.com/page)

I'd also add that you can of course also have different content even when the URL and protocols are exactly the same. For instance https://example.com may display different content to different users depending on factors such as when they're accessing the page, where they're visiting from, the device they're using, etc.

The upshot of this to the OP is that from a technical perspective you cannot know with 100% certainty whether it's worth scraping both http and https versions of what are otherwise the same URLs just by looking at the URLs. So it may be a good idea to eyeball some of the pages in question to see if they are different, and if so if their differences are meaningful enough for you to include both.

That said, in 2020 https is more or less taken for granted by any decent website, so in practice I very much doubt you'll see much -- if any -- significant differences between the http and https pages. However, I do think you might notice some meaningful differences based on some of the other factors I mentioned above (time, location, device, etc.)

Edit: Just noticed the question about including or excluding the www in a URL.

The answer is pretty much identical the the one about http/https. Technically, the content could be different between two URLs differing only in whether they include www (e.g., https://example.com or https://www.example.com). So again, you can't know with 100% certainty just by looking at the URL alone and thus eyeballing a few of the sites in question is probably a good idea. That said, by convention pages with or without the www are almost always identical, so you'll probably be able to get away scraping just one or the other.

One thing I should have mentioned above is that some websites include their canonical URLs in a tag like this: <link rel="canonical" href="https://www.example.com"> (see https://yoast.com/rel-canonical/). If one of the sites your scraping uses these that may indicate that you'll only need to scrape those URLs.