It's well known that Google tends to favor user generated content, so Wikipedia, StackExchange and so on often come up as the top results. However, I've noticed that I rarely ever see Reddit or Hacker News showing up despite the quality discussions going on over there. Is there any reason for that? Is it due to poor SEO? In general, what could trigger Google to generally ignore these two websites?
You're basing that blanket statement on what?
There's also a distinct difference in purpose. Reddit and Hacker News pages are basically chat rooms with an outbound(usually) link at top, without necessarily much actual or focused information about the thing they're about in the page. (You click over to the original URL for that.) I'd imagine there's not too much for a search engine to grab onto for assigning relevance. They don't necessarily rank badly, but other than the occasional case where they break some information and get heavily linked I don't see too much reason for them to rise all the way to the top of results.
As one practical example, this story about Rocket Internet is near the top of Hacker News right now, posted two hours ago as of this writing. It's not all the way at the top of a Google search for "rocket internet"(for me, since all searches are customized now), but it is near top of the third page of results, which is fairly impressive given it's only two hours old.
As far as my comment about not always using the same ideas of "authority", if I do a search for "refrigerator" the top results are Sears, Best Buy, Loew's, etc which are clearly not user-generated sites, and not even especially informational. This is because Google presumably determined at some point that a person doing such a search is probably looking to buy one and not just read about them.
Agreed with the other posters re: Google not being so much in favor of user-generated content as they are authoritative content. There's another factor that revolves around this same principle that comes into play here: backlinks. Although it's now one of a number of ranking factors, Google still equivocates quality backlinks with authority.
Reddit and Hacker News both have some good, meaty discussions, true. But both are inherently transient sites - meaning, a discussion will be linked from the HN/Reddit homepage for a day at most, generally an hour or two at best. There's no categorization (at least for HN), & no 'staying power' for each story posted...so each story posted tends to not become an authority on issues, in terms of # of backlinks.
Additionally, the topic (H1) for each Reddit/HN page describes the page it links to - it doesn't describe the discussion at all. And the link to each discussion has the anchor text 'comments' (for HN - I can't rember what it is for Reddit), which does nothing to differentiate that page from any other linked page on the site.
StackExchange sites, on the other hand, are categorized, crosslinked, and very searchable...which leads to each question having the potential to become an authoritative page. It's much more likely that someone will link to a 'How do I accomplish X?' question on StackExchange than they will a 'Check out this cool link' discussion on Reddit or HN.
So in essence, the very nature of the StackExchange sites (Q&A, rather than link-sharing) - as well as the indexable, categorized structure of the site - make its individual pages much more likely to appear at the top of the SERPs than an unfocused discussion on Reddit or HN.
Google don't like link farms and sites with non-original content scraped or recycled from other sites. I believe both the sites you mention as Google non-favourites fall into those categories and is therefore (IMHO justly) penalized by Google.