I'm wondering the best way to handle this situation. I'm about to release a new section on a website where there are literally hundreds of millions of URL combinations that can possibly be crawled and indexed by search engines. This is because I'm giving the users the option to choose their refinements using checkboxes, similar to Zappos, and there are tons of possible combinations.
The website currently has about 100k URLs indexed by Google but releasing this new section of the website will increase that number by a very large magnitude. This could be incorrect, but I've heard websites may be penalized for having a huge sudden increase in crawled pages. Is this true? If so what's the best way to handle this? My goal isn't to "pagerank sculpt" or anything like that, but I don't want to be penalized for any changes I'm about to make.
I thought about adding a "disallow" directive to my robots.txt file, but that will block out the indexing of all of the new pages which I don't want to happen. I think it's best to have the most important ones indexed (one or two filter combinations), but I don't think it's really necessary to have the extremely long-tail combinations indexed. So I guess my question is: is there any good way of doing this or is it best to just let the spiders to their thing?