Google does not always inform site owners about penalties. Some will argue, but there is a reason why the Google Webmaster Tools (GWT) lables this option as Manual Penalties. In the beginning that's exactly what that feature was. But now Google has automated several additional penalties but not all. So please do not count on this feature as the definitive proof that you are not penalized.
From what I have seen over the years, there are several reasons why search performance can drop due to links.
- Sudden increase in links makes Google nervous. (temporary)
- Unnatural back linking (repetitive/manipulative).
- High ratio of low quality back links.
- Participating in a back link scheme.
- Negative SEO.
- Toxic back links mostly due to spammy or low quality sites.
(temporary)
This is just off the top of my head. There may be more I am not considering here.
I suspect you fall into the first and/or last items in this list. In the first case, Google is just being cautious while they reassess you link profile do to a sudden change. In the second case, most of these sites can disappear and change link strategies very quickly allowing for recovery automatically. This is not necessarily a given and should not be relied upon.
If you have a popular topic, the various spammers and content scrapers will take notice and try and bleed off any search success for themselves. This has been going on for may years of course. And while Google is combating this, these tactics still work for a broad spectrum of search engines and sites.
You can use GWT to extract a list of links using Links to Your Site and More. From there, you should be able to download the links that Google is aware of as a CSV (comma separated value) file. It has been a while since I have done this. You can open this file in any spreadsheet.
I found this excellent list of things to look for here: http://www.audiencebloom.com/2014/01/perform-full-backlink-audit-remove-toxic-links/
- Obviously spammy sites, porn sites, payday loan sites or gambling
sites
- Sites that are not indexed by Google
- Sites with a virus or malware warning
- Sites that have no Page Rank
- Sites with very new domain names
- Sites with very little traffic
- Link networks
- Sites with an unusually large number of external links
- Irrelevant sites
You can use your spreadsheet to: one, sort by URL, two, code each site according to category. From there you can determine what sites really should be disavowed.
For the rest, you may have to risk visiting the site/page to see for yourself what is going on. But do this with some caution of course.
Keep in mind, you want to disavow links from a site in a conservative manner. DO NOT get too carried away. Just hit the most toxic sites first if at all and see what happens. You will need to use your judgement of course and detach yourself from any emotion you feel.