In it's very basic form search engines don't crawl content provided by ajax. Google at least gives a lookout on this matter:
In the near term, your site will remain indexed by Google as-is, with many pages likely not fully represented in search results. However, we are continously working to make Googlebot behave more like a browser. As we implement more features, Google may start to index your pages properly without help.
So it's not possible now but maybe in the future. However, there are ways to make ajax crawlable (e.g. HTMLsnapshot) which you can read about here: https://developers.google.com/webmasters/ajax-crawling/docs/getting-started
So back to your questions:
So is there a trade off with lazy loading?
Yes, without some changes this content won't be indexed (unless it is also statically available on your page somewhere, crawlable with the above mentioned changes or available through a sitemap). This obviously can result in a lower ranking (based on the loaded data).
The site will download faster, but loses some optimization?
Not necessarily. The ajax script is still getting executed and loading in the background. The overall load time could be even slower. Lazy load the content that seems to be secondary to the user (there are exceptions to this of course, sometimes even primary information makes sense to load this way)
Do you just need to be careful what to choose in the loads?
That's basically it. Profile the performance and try to figure out if this enriches the user experience.
Once again I can only recommend https://developers.google.com/webmasters/ajax-crawling/ where you can also read about What the user sees, what the crawler sees
