I want to maximize the size of the screenshot and avoid scroll bar or overflow for the maximum too.
What size do you recommend ?
I'm using lightbox.
Here's the excellent Browser Size tool from Google showing how many percent of users will see the different areas of your website. This really helps to check webpages and ensure that people on 'shallow' screens (most laptops have very wide but very shallow/not deep screens) can see your important content and calls to action.
Update: Google has now added this functionality to Google Analytics, allowing you to see how much your visitors can see of your site: http://analytics.blogspot.com/2012/06/new-feature-conduct-browser-size.html
This could vary by site and user base, but many websites are still supporting a screen width of 1024 pixels wide (because there is still a sizable user base using that resolution). Therefore, you'd want your image to be big enough to fit into a space that was 1024 pixels wide. Let's say you have 15px of chrome on the left and right sides of your lightbox and you wanted a 30px margin on either side of that for people running 1024px wide monitor resolutions.
1024 - 30px(lightbox chrome) - 60px(margins) = 934px wide image.
Again, the specifics might be different for your site, particularly if you want the lightbox edges to lay at a certain point in relation to your page or if you are trying to snap to a grid or something like that.
A cool solution would be to upload the screenshots at a max resolution of 2560x1600. Then when the page loads, detect the browser window size, and use an image handler to resize and save the image on your server on its first request.
That way people with high resolution monitors don't get neglected.