I manage a website which is in fact a search engine for a specific branch of industry. Because most of the sites are dynamically accessed via form (where a user can input a keyword to search for), I thought of submitting a sitemap to Google to inform their crawlers about my pages.
There are two problems:
- I have 25 million pages (and I think the limit per sitemap is ~50 MB, which will be exceeded)
- I want to remove the sitemap after submit, because I did not want to be crawled by competitors
Should I submit a sitemap to Google or is it better to let the crawlers index the page by themselves?