The objective measure is in money: How much does it cost to maintain the browser support vs how much it costs to turn away users because of browser compatibility.
Costs of turning away users
- Immediate revenue lost from sales or advertising
- "Bad will" where users remember that your site doesn't work and are less likely to use it in the future
- Wasted customer acquisition costs
- Impacts on SEO from bad usability or "mobile friendly" scores
Costs of adding browser support
- Developer time (which may be difficult to estimate ahead of time)
- Quality assurance time
- Equipment and access to browsers
- Opportunity cost (you could have been working on something else that makes more money)
- Not being able to use technology because it is unsupported by some specific browser. (This may make your site look or behave worse for everybody.)
Having evaluated these factors myself, I usually use a 2% threshold. If the browser has at least 2% market share, it is worth supporting.
For the last 10 years, old versions of IE have been the most costly to support. They typically require more workarounds and support fewer features than other browsers. They also tend to be used far longer than old versions of other browsers. The costs of supporting IE 8 (3%) could be twenty times the cost of adding support for Opera (1%). You sometimes have to take it on an individual case by case basis and see how badly your website is broken in that browser.