Can this impact my site negatively?
Not usually, for two reasons.
One, Google is very aware when a site is using WordPress and the algorithm is smart enough on its own to know the difference between content displayed on a category or tag archive and on the post/custom post type itself. If you really want to be paranoid about this, create custom excerpts for all content and set your archive pages to display only the excerpt instead of the full content and that will make you safe beyond all doubt. This is overkill, IMHO.
Also, if you use an SEO plugin (Yoast, Platinum, All-in-One) that automatically generates the canonical URL and outputs it for you then you really won't have anything to worry about.
That being said, categories and tags can impact you (positively and negatively) by generating confusing navigation (bad UX) and potentially appearing spammy to search engines if you overdo it.
Would it be better to not use tags at all in an attempt to prevent duplicate content issues?
As above, I doubt that duplicate content would ever be of major concern given a normal attempt to classify various posts for your visitors. I would be more concerned about overdoing categories and tags because that's what people do in the attempt to Classify All The Things. For that reason, I personally tend to not use tags on most sites I control unless there is a clear and limited list of tags agreed upon by both the SEO team and the content generators.
