Without the plugin that generates them, the sitemaps won't become invalid, but they won't get updated either. New pages won't get added to the sitemap, deleted pages won't get removed from the sitemap, URL changes won't get updated in your sitemap.
One of the biggest advantages of sitemaps is that when you add a new page or post to your site, it gets added to the sitemap and Googlebot comes to crawl it quickly. If you are not adding new pages to the sitemap, Google will find the new pages by following links on your site, but it may not happen as quickly as with a sitemap.
Another advantage to sitemaps is that they tell Google which URLs you prefer. If you change URLs for some pages and don't update the sitemaps, you will be giving Google conflicting signals about what the correct, canonical URLs should be. Google is pretty good about choosing one or the other, but Google may not always make the choice you would want if your sitemap is out of date.
You probably don't need an XML sitemap for SEO, they don't improve rankings and have little impact on which pages get indexed. However, if you are going to have a sitemap, you should keep it up to date. I wouldn't recommend deleting the plugin that generates your sitemap.