When should I choose CDN and when is it a good idea to use your own server? Would using your own server be more cost effective?
It may be - if you have a high and steady stream of traffic then your own server may be more cost effective. CDNs however may let you spike in traffic, or adjust what you pay on the fly according to demand, so they may be better if you have very low traffic or if your traffic is subject to change or spike.
This of course varies between providers, so keep an open mind when shopping around.
Aside from price, there is also the question of whether you are competant enough, or can afford the time, to set up your own server and keep it secure, backed up and up to date.
How are CDNs integrated ussually? Is there a client software installed on my server that pushes new data (user uploaded content) to the CDN?
This varies between providers, and I have no experience with the bigger ones (Amazon, etc) to answer this bit in depth.
What is the best practice for setting up your own static server and is rsync used for moving files over?
Yes, you can rsync where-ever SSH is supported and it's how I usually move files to a server.
I find the Linode library a great place for resources about setting up a linux based server.