There is some damage that this can cause to your site. The amount of damage that it can cause ranges from extremely small to a moderate amount.
Anything that a webmaster does to his webpage that isn't quite right or diminishes the user experience can hurt your rankings on Google. Google's algorithm is designed to give its customers the very best experience that it can in its results. A website that is about "cars" but has a link to "smurfs" on the page is in itself a poor user experience. The page about cars could be amazing, 100/100, but the link to smurfs might make it more like a 99/100. Because the link just doesn't really make sense.
On top of that, a lot of SEO experts have found that Google values links based on their association. It's vital to be linking to content that is relevant to a page. When a page about cars links to a page about smurfs, the page relation doesn't make sense, and to Google's algorithm, this could look slightly spammy.
Google has become an expert in understanding words and language. And so it knows what the word "smurfs" is. It knows that we're talking about little blue cartoons. If that doesn't have anything to do with the on page topic, then it's off topic. And Google analyzes anchor text to determine what the following page is about.
If you're internally linking to a page about biology with the anchor text of smurfs, Google will likely see this as spammy anchor text. And Google has taken serious measures to stop webmasters from abusing anchor text for years. It's taken such a serious approach against it because websites used to rank their dirty/spam sites for keywords such as cars by flooding car related anchor text to all of their backlinks. Tags are generally used as internal links with anchor text.
Google wants to show users a page about cars when it searches for cars. If the user sees a link to smurfs, the page is slightly less about cars than it was before.
By messing around with the anchor text in the tags on your site, you're almost certainly going to be slightly damaging it.
One little tag link to an off topic of "smurfs" may not actually change your rankings, but the concept of it is actually dangerously running the risk of getting your site hit, especially if the tag structure of the site is misused many times.
Sites that are built with the absolute best intentions for user experience are most likely to rank. And Google is getting better and better at understanding when a user experience was positive or negative. Messing around with on page content that doesn't belong, or using improper anchor text for your tags is intentionally sabotaging your own user experience.
Overall, one little incorrect tag is probably no big deal. But messing around with things like that could cause more harm than good.