The truth is Google wants you to succeed.
Luckily, Google in most cases is fair and accurate in its ranking of those who get penalized and at the same time rewarding those who are not using any deceptive practices.
Everything affects SEO and Traffic Exchange programs.
But how you are involved in it makes all the difference. Are you condoning misbehavior, using tactics that are obviously deceptive and manipulated or is it things out of your control. (Things out of your control still should be paid attention to as you will see further down.)
Should I be worried about how people link to or enter my site?
If you know for a fact that you are not giving incentives for people visiting your site, not exchanging irrelavent content links and keeping your content relevant to links you publish in your content than you will be fine.
What are some things I can do if I see sites pointing to my website and I never asked them to?
First locate your backlinks (you can do that in Google Webmaster Tools) and determine if they are quality links. If they are links from trusted sites and relevant to your content then great! If not you can contact the site owners and ask them to remove the backlinks and if not, Google does offer a way to disavow a link to your site. So in turn you are telling Google that hey, I do not want the link to be associated with me.
I only share links with other site owners and not traffic exchange programmes, is that ok?
IF the site is relevant to your content or users do in fact find relevance in sites that appear to be not relevant.
You should not have a site about Tires and exchange/post links to a site about Pets. (Unless it's in an AD form and you make it clear that it is such. IE. Support Your Local Pet Rescue)
The same principles apply to those linking to your site.
A good example of links appearing to be irrelevant, would be a site A, a store that sells Jewelry and has a back link to a Site B, a Floral shop. That is perfectly find because people do tend to buy flowers along with jewelry for instance an upcoming wedding.
I do not practice deceptive marketing, what else can I do?
Only thing at that point to have concern over is things you can't control, for instance you have been hacked and someone has injected redirects, in the form of hidden spam links or malware of any sort.
It seems you have GA and so they will spot and notify you if that is the case. Also you want to check on your backlinks, which can be done on your GWT, and those you spot, take action on. You can do this by contacting the site owner or blocking incoming clicks from the site.
In the End as in the beginning - Google wants you to succeed.
They have many things in place to spot site owners willfully involved in user behavior manipulation and will penalize those sites and I don't mean Call to Actions, that is good marketing!
Keep this in mind, you can easily spot when a website's link has tricked you upon entrance of their site and so it should be easy for to know if what you are doing is not benefiting a user's desire.