If your content was created using page building plugins, your website will break if you remove them, so that is not an option. Yoast SEO is essential. If nobody can find your website, it will not matter how fast it is. Without the benefit of seeing the website in question, I can only give very general advice.
As a WP freelancer, I have seen many times when jQuery or other JS libraries were loaded multiple times. You can load the page in any browser and press Ctrl+u to check for these types of problems. It is often the first thing I do when diagnosing a slow WP website.
If you are loading large, hi-res images, they need to be run through a lossless compressor, like compressor.io.
I have found the WP caching plugins to be a huge curse and I do not use them. When a WP website goes down, one third of the time it is the caching plugin. Instead, I suggest that you use cloudflare.com. One really needs to identify the problem and resolve it instead of adding problematic caching plugins.
Honestly, there may not be a single fix, only 100 smaller things that you could do. You could move the whole website to a specialized WP hosting provider like wpengine.com, however, CloudFlare can do most of what they are doing for free.
If only the home page is slow, then your server may be fine. That would indicate that you have too many plugins, huge images, or problems like I described in paragraph 2 on the main page.
To answer your question about loading JS in parallel, WP doesn't make it easy to make these changes. It is much better to combine all JS into one file but it is a very technical job to do it correctly and without breaking the site.
I am a big fan of gtmetrix.com. If you are getting a low score, it may be time to call a WP professional to help resolve the big issues.