The objective measure is in money: How much does it cost to maintain the browser support vs how much it costs to turn away users because of browser compatibility.
Costs of turning away users
Immediate revenue lost from sales or advertising
"Bad will" where users remember that your site doesn't work and are less likely to use it in the future
Wasted customer acquisition costs
Impacts on SEO from bad usability or "mobile friendly" scores
Costs of adding browser support
Developer time (which may be difficult to estimate ahead of time)
Quality assurance time
Equipment and access to browsers
Opportunity cost (you could have been working on something else that makes more money)
Not being able to use technology because it is unsupported by some specific browser. (This may make your site look or behave worse for everybody.)
Having evaluated these factors myself, I usually use a 2% threshold. If the browser has at least 2% market share, it is worth supporting.
For the last 10 years, old versions of IE have been the most costly to support. They typically require more workarounds and support fewer features than other browsers. They also tend to be used far longer than old versions of other browsers. The costs of supporting IE 8 (3%) could be twenty times the cost of adding support for Opera (1%). You sometimes have to take it on an individual case by case basis and see how badly your website is broken in that browser.
Looking at the data, I would only support IE8 and IE9. However, IE8 may be lacking some functionality which may be core to your site, so you have to weigh up the benefits versus the time spent to support it.