Why wouldn't the URL be case sensitive?
I understand that may look like a provocative (and "devil's advocate") type of rhetorical question, but I think it's useful to consider. The design of HTTP is that a "client", which we commonly call a "web browser", asks the "web server" for data.
There are many, many different web servers that are released. Microsoft has released IIS with Windows Server operating systems (and others, including Windows XP Professional). Unix has heavyweights like nginx and Apache, not to mention smaller offerings like OpenBSD's internal httpd, or thttpd, or lighttpd. Additionally, many network-capable devices have built in web servers that can be used to configure the device, including devices with purposes specific to networks, like routers (including many Wi-Fi access points, and DSL modems) and other devices like printers or UPSs (battery-backed uninterruptable power supply units) which may have network connectivity.
So the question, "Why are URLs case-sensitive?", is asking, "Why do the web servers treat the URL as being case sensitive?" And the actual answer is: they don't all do that. At least one web server, which is fairly popular, is typically NOT case sensitive. (The web server is IIS.)
A key reason for different behavior between different web servers probably boils down to a matter of simplicity. The simple way to make a web server is to do things the same way as how the computer/device's operating system locates files. Many times, web servers locate a file in order to provide a response. Unix was designed around higher end computers, and so Unix provided the desirable functionality of allowing uppercase and lowercase letters. Unix decided to treat uppercase and lowercase as different because, well, they are different. That's the straightforward, natural thing to do. Windows has a history of being case-insensitive due to a desire to support already-created software, and this history goes back to DOS which simply did not support lowercase letters, possibly in an effort to simplify things with less powerful computers that used less memory. Since these operating systems are different, the result is that simply-designed (early versions of) web servers reflect the same differences.
Now, with all that background, here are some specific answers to the specific questions:
When URLs were first designed, why was case-sensitivity made a feature?
Why not? If all standard web servers were case-insensitive, that would indicate that the web servers were following a set of rules specified by the standard. There was simply no rule that says that case needs to be ignored. The reason that there is no rule is simply that there was no reason for there to be such a rule. Why bother to make up unnecessary rules?
I ask this because it seems to me (i.e., a layperson) that case-insensitivity would be preferred to prevent needless errors and simplify an already complicated string of text.
URLs were designed for machines to process. Although a person can type a full URL into an address bar, that wasn't a major part of the intended design. The intended design is that people would follow ("click on") hyperlinks. If average laypeople are doing that, then they really don't care whether the invisible URL is simple or complicated.
Also, is there a real purpose/advantage to having a case-sensitive URL (as opposed to the vast majority of URLs that point to the same page no matter the capitalization)?
The fifth numbered point of William Hay's answer mentions one technical advantage: URLs can be an effective way for a web browser to send a bit of information to a web server, and more information can be included if there are less restrictions, so a case sensitivity restriction would reduce how much information can be included.
However, in many cases, there isn't a super compelling benefit to case sensitivity, which is proven by the fact that IIS typically doesn't bother with it.
In summary, the most compelling reason is likely just simplicity for those who designed the web server software, particularly on a case-sensitive platform like Unix. (HTTP wasn't something that influenced the original design of Unix, since Unix is notably older than HTTP.)