this is definitely possible, but in the last month, Google did have a pretty significant algorithm update focused on "Mobile Speed," as part of it's "Mobile First Indexing" shift, and I would say that it's possible your organic performance has dropped more likely to that.
Also, what are you using to track your backlinks? With our clients, we use about 3 or 4 tools to track and monitor all of their backlinks, as it's hard for one tool to get absolutely all of them.
If you haven't already, make sure you're signed up with Google Search Console for your site. Check and see if the links are aligned, or if there's anything you missed. Also, check out the new dashboard and see if there's anything under the Mobile Usability Errors section. They've just updated that and it can provide some pretty useful insights.
Another thing I should say, there are hundreds of ranking factors, and likely thousand of "sub-ranking" factors, such as the anchor text of a link, the text surrounding a link, and under content, we have the semantic element, the relevancy, etc. And then of course, you have competition. It could be that some of the competitor sites that you're competing against for a top spot in the rankings, either did something differently or reacted better to Google's recent algorithm update.
This is a pretty short summary answer to your question, but I hope it helps. Deep dive with a few other tools such as Majestic, SEM Rush, Ahrefs, Spyfu, etc. Def make sure you know Search Console in and out though.
Also, it may help if you sign up with Bing webmaster tools - I mean it's free so you may as well, if you're not already. If I'm not mistaken, it'll also show you some links and has some pretty useful features (And there are people that do use bing, so good to know/be familiar with). Good Luck!
Here are some resources and info on the recent Google updates such as Mobile First Indexing and Mobile Speed:
- Announced officially in March, but was slowly rolled out mostly in the later half of the year. Here's Google's announcement on it.
- Websites got notified individually via Search Console when their site was enabled for Mobile First Indexing
- Mobile Speed was heavily emphasized in the summer, with Google advocating sites focus on Mobile Speed, and we kind of had the impression the update would focus on that (which it did, slow sites were penalized), but it really was kind of an overall integration of all Mobile First Indexing, I think. Speed is of course an important factor now.
Some more links:
Also, this is quite interesting:
Just the other day, I came across this comment on Reddit from John Mueller of Google (like the SEO guy, or the Google Engineer that actually gives some insights to SEOers and whatnot on how Google's algo works..), where he actually says, in so many words, that Googlebot only looks at the mobile version of the site now for rankings. This was a bit of a definitive statement, compared to before, where Google was saying things like "primarily mobile", etc. You can find the comment here.
In other words, Googlebot is rarely if ever crawling the Desktop version of most websites now, according to this statement.