HTTPS has more overhead, mainly at the connection setup, and is thus slower. See https://security.stackexchange.com/questions/64825/is-there-ever-a-good-reason-not-to-use-tls-ssl/64832#64832 for more technical details.
Thats why the most important optimization is to reduce the connection overhead by both reducing the number of connections and by reducing the overhead of each connection. Without knowing your site I would recommend the following things:
- Do not include any third party sites. The more different sites are involved, the more connections need to be created and the more the overhead of HTTPS for a single connection slows things down. Especially don't include any advertisement networks, where the final ad is not served directly but only after several redirects between lots of different hosts.
- Reuse SSL sessions. This needs to be configured at the server side (most have it enabled). It might also be, that your load testing does not support session resumption and thus everything looks slower as it is in reality.
- Use keep-alive, that is multiple requests over a single TCP connection. This reduces both the overhead from connection setup for HTTP, but even more for HTTPS. Tune the relevant timeout at the server so that the server does not close idle connections too fast.
- Offer ECDHE ciphers, like most TLS1.2 aware servers do. Both ECDHE and DHE key exchange provide forward secrecy (compared to RSA key exchange) but DHE is really slow, while ECDHE is not much slower than RSA key exchange. This is relevant for session setup only. TLS1.2 also introduced some new ciphers (GCM) which offer more performance with less load.
- Dedicated SSL accelerators or load balancers will only help if the problem is the load on the server. Otherwise they just introduce more overhead and might even slow things down because some only offer sub-optimal ciphers.
And of course any optimizations you do to speed up HTTP sites work also for HTTPS sites.
Edit: a comment from @jeffatrackaid mentioned SPDY. This is a very good proposal since SPDY also reduces the number of connections further by putting multiple requests inside a single connection, but smarter than HTTP keep-alive (not one after each other, but multiple interleaved). And SPDY is supported by Google Chrome for a long time, Firefox too for some time and I think the newest versions of MSIE have it too.