If they're only appearing in website traffic reports (like Google Analytics) and not under Links to Your Site in Google Webmaster Tools, then they're likely not going to impact your SEO - just don't click on them.
If they are appearing under Links to Your Site, then spammy and low-quality incomming links might make a difference. From Google Webmaster Tools:
Google works very hard to make sure that actions on third-party sites
do not negatively affect a website. In some circumstances, incoming
links can affect Google’s opinion of a page or site. For example, you
or a search engine optimizer (SEO) you’ve hired may have built bad
links to your site via paid links or other link schemes that violate
our quality guidelines. First and foremost, we recommend that you
remove as many spammy or low-quality links from the web as possible.
If the appearance of incoming links seems to correlate with the decline, it might be wise to disavow them:
If you’ve done as much work as you can to remove spammy or low-quality
links from the web, and are unable to make further progress on getting
the links taken down, you can disavow the remaining links. In other
words, you can ask Google not to take certain links into account when
assessing your site.
First collect and download the links to disavow according to the steps here, being sure to only include those in the list that you want disavowed. Alternatively you can disavow all links from an entire domain by adding
domain: example.com to the list.
Next upload the list to the Disavow links tool.
As Google also recommends:
You should still make every effort to clean up unnatural links
pointing to your site.
Try to contact the operator of the referring site(s) to request that they remove links to your site, and/or block them with your web server configuration.
Since your site is small, it might help to continue adding relevant content to it and obtain backlinks from authoritative relevant sites.