The old common knowledge was that you can't get penalized for sites linking to you. It was true at that time. Post-Panda, you can get sites penalized a lot easier with certain linking patterns and, yes, it can be used to penalize weak competitors. If you want to see this in action and it's worth spend $5, find a ranking site with only a few weak links and buy a Xrumer blast for it at Fiver. Odds are it will drop like a rock.
Note though that this applies to sites with a weak link profiles and perhaps only for certain keywords. You aren't going to do this with Amazon or CNN but with Joe Blow's pink dog collar niche affiliate site it works.
I'm not saying a competitor intentionally did this to you. It could have been someone testing SEO pattern changes or somebody using your site for link camouflage. The bottom line is that getting rid of the links won't be easy if you can even do it at all.
The solution is to work on building better and stronger links to your site. This will make your site less vulnerable to such tactics.