RAM usage by Linux
Linux will generally eat a lot of RAM to avoid disk caching (swap file) and does not indicate that anything is wrong if you have little to no ram free. You should only consider it a problem when you have no free memory and your Linux OS is using the swap file.
Linux Swap File
Disk Swap caching should be avoided as much as possible since SQL queries that occur on disk are much slower and cause the system to start queuing requests, this increases CPU usage and IO read/writes.
It's a common question by Webmasters do I need more RAM or CPU? the answer is neither one or another, and varies from site to site depending on the processes, if they are more SQL based, or CPU based. SQL will hit both CPU and RAM, but if the SQL is cached, it does not hit the CPU as much since its processed fast and then goes on another. In most cases, more ram will cure the resources issue.
If your server is using the swap space then you need to increase RAM. Most sites will generally need more RAM than they need CPU, so you could always reduce to 1 core and increase the RAM. Personally, given the choice I would choose 1 CPU core 1.5GB RAM over 2 CPU cores, 1GB RAM.
Vanish HTTP Cache
Varnish is an HTTP accelerator designed for content-heavy dynamic web
sites as well as heavily consumed APIs. In contrast to other web
accelerators, such as Squid, which began life as a client-side cache,
or Apache and nginx, which are primarily origin servers, Varnish was
designed as an HTTP accelerator. Varnish is focused exclusively on
HTTP, unlike other proxy servers that often support FTP, SMTP and
other network protocols.
Varnish is used by high-profile, high-traffic websites including
Wikipedia, online newspaper sites such as The New York Times, The
Guardian, The Hindu, Corriere della Sera, social media and content
sites such as Facebook, Twitter, Vimeo, and Tumblr. Of the Top 10K
sites in the web, around a tenth use the software.
A lot of 'GOOD' Hosting companies such as GANDI use Vanish, its incredibly fast and has been known to speed up sites by 100-1000x faster.
Content Delivery Network
You can also reduce the amount of fetches by using a CDN network, they are very affordable and will help you reduce server load but also... help you speed your site up since they will load the nearest file to the location of your visitor. Google rewards for fast sites and I recommend Cloudflare and similar services.