One of the things I've found like your roofing example is that sites with a relatively poor quality pool of backlinks outranking our site for certain competitive terms even though we have a much stronger pool of quality backlinks. What I've found is that with links from 'proper sites (news sites, larger blogs etc)' they tend to use your company's name as anchor text rather than a keyword.
Some competitor's sites tend to have a lot of keyword-rich anchor text pointing back to them, even though the links are from sites which are less than quality - think something like bestonlinearticlebase.com (that's just a made up example but you get the idea.)
By far the best tool in my opinion to see their anchor text profile, is Moz's Open Site Exploer; enter the domain of your competitor's site (or yours to see how you fare1) and then click under the anchor text tab to see a break down of the anchor text, and the sites it's coming from. Another really useful tool in there is the Linking domains tab in which you can see all the domains linking back to them, which is by default sorted by Domain authority & Page authority (similar to Page Rank). All in all it's a really useful tool for getting your mind around a competitors strategy, definitely worth a look through in detail.
What's interesting is that the Penguin update was meant to wipe this sort of stuff out and to some extent it does, i.e. sites with too much keyword-rich non-brand-specific anchor text do get penalized, but in my opinion especially from the sites I've seen outranking us, Google haven't turned the settings up strong enough yet.
From my experience and digging through the Open Site Explorer data, relevancy is less important than authority, but of course the balance tips if you had say a PR4 relevant site, and a PR5 non relevant site (I know PR is a pretty nasty metric but you get the idea.)
All this being said if you mimic a competitor's bad practices for short term gains and Google changes what it sees as white hat and clean, you could be hit too, so proceed with caution .. it's a real conundrum though when you are being outranked by bad practices..do you join them, or hope they get caught but lose the traffic in the mean time...