Would you be able to post part of your httpd.conf (or included configuration if it's split into multiple files)?
Inside of it should be the "per user" configuration section, which defines which directory the servers looks for in a user's home.
For example, here is the default I normally have on my Apache servers:
# Turn on userdirs
Allow from all
In this case as long as a user has a public_html directory, with permissions of 0711 (or at least 0755) - the read and execute bits of public_html must be set, along with at least the execute of their home directory - Apache will serve files from it with the proper permissions.
One other question, from your URL above of "server/~user/public_html/" - did you upload the files under the directory public_html/public_html? Otherwise, you should just drop the "public_html" part at the end, as Apache will automatically look under there.
That would make the URL http://server/~user/ and should work, unless your server has a non-standard configuration.
If none of the above works, check to see if you have directory indexes enabled, and remove any index files from your public_html; that should give you a list of all the files, to verify the structure!