While there are differences, there is not much that cannot be made compatible in older browsers depending upon what you are doing.
If you are developing your site and wanting to add features but feel hampered by the lack of advance in the use of newer versions of a browsers, then you have to decide what you are really doing. For example, if you are developing an audience to sell product or to make money through advertising, then really, you have to meet your users where they are. To optimize a site can mean remaining intentionally simple and doing the added work to make sure you are serving as many potential customers and users as possible.
However, if you are developing a website that is an application or technically involved, that could be very much different. For example, I once worked on a reporting engine that had three elements; static "canned" reports offered "ala carte", dynamically defined "ad-hoc" reports, and cubed-data analysis, of various services already purchased. Each level of access had a cost and each level of service had a technical requirement that could not be avoided. In this case, in order for the user to access the service data they had purchased, the technology required that the browser be at a minimum level before these services could be accessed. In this case, prior to purchase, these requirements were made quite clear. The advanced use of technology was the price of entry for advanced services.
If you are developing a website and you want to do certain things and do not want to support older browsers, of course that is your choice. If your site does certain technical things that require newer browsers, of course this is your choice. Keep in mind that the narrower you make the price of entry to your site, the fewer people you will be able to serve.
It is a numbers game really. What is your goal? Is it necessary and reasonable to implement a certain level of expectation from your user? And would you miss your goal metrics if you implement a standard that would limit the market? It may be penny wise and pound foolish to skip the added development effort to accommodate users of older browsers. Only you can decide that.
But here is what you will miss. I have been to these sites that ask me to update my browser. I use Chrome and it is up to date or reasonably so. The request in of itself interrupts the user thus effecting the user experience (UX). Often a user may not be in a position to update. It is perfectly reasonable that a Windows XP user has IE 8 and will visit your site. If you ask them to update their browser, you have asked them to spend hundreds of dollars that they likely feel is unnecessary. It may be an older computer, but not so old as to be replace it yet. Maybe what they are searching for exists on your site but the user can get the information elsewhere. I cannot tell you how many times this has happened to some of the people I deal with and the result is always the same. The user skips your site and goes elsewhere.
I would not recommend asking a user to update the browser, unless there is a clear technological advantage to the user and only after I have made it clear that the offering you have requires technology that exists in newer browsers.