What are the differences of the two and how can they help in SEO? I'm creating a custom-scripted CMS and I'm thinking if should I either add RSS/Sitemap/Both?
RSS helps users browse your site in an aggregated format without having to visit the site. There is some evidence to suggest that feeds help keep your site 'freshness' up in search engine rankings so they are worth the effort, but how much this affects your ranking is anyone's guess.
Sitemaps helps search engines index your site and better understand when new content has been added or updated. IMHO these are a vital part of any site, if for no other reason than it keeps your crawl rate down - I doubt they contribute much towards SEO in the traditional sense.
Any CMS should offer both IMHO.