You've clearly been penalized, thus the fall on traffic. A whole day with the server down its a lot of time.
For sites from about 2000 pages Google crawls some pages of the site almost every day for changes, several times a day. If no changes are found in time, it crawls them less frequently. So, all of the crawled pages in that day have changed from quality content to 404/500 page content.
Indexed 404
's and 500
's are really bad for SEO as they get very penalized by Google, even more if they are not informative (soft) error pages.
First of all, make sure you've fixed your server problems, you don't want 500
's to get crawled again.
As time passes, your pages, will get back near to their original position, some in days, some in weeks, some in months.
A laborious trick is to change your site's appearance/distribution, Google will crawl way more pages a day than it does normally to keep the site fresh in SERPs.
Another thing you can do is to add your sitemap.xml
(if you haven't done it already), Google loves them and may crawl the pages in it for the first time earlier than normal.
Ps: site:mysite.com
is not updated in real time. It's just useful to give you an approach of your site's index in Google.