I was doing some link-building, forums,blogs,social bookmarks,directory submission <-- that's most likely your problem, you've been hit by Google's evil penguin and yes it's very common for those who create links themselves as Google regards this as attempting to manipulate search results.
Non Relevant Links
Google now punishes sites for links from non-relevant sites on a big scale, these include but not limited to, forums reply's, forums signatures, blog comments, low quality social bookmarking sites i.e pligg, directory sites etc etc. Lot's of questions and answers regarding penguin on the site.
You should also note that most recently millions of sites that were affected by Penguin have had their punishments lifted, so you might not be affected much so to speak but rather that people have got their rankings back and knocking you down.