How should I create sitemap of a site which goes live with thousands of pages?

Question

We are working on an education site and have custom website built on wordpress. One component of site is blog posts and pages, sitemap for which gets built by sitemap plugin every night.

Another component is the college pages and we went live with this section 2 month back. To build the sitemap for these 8000+ pages , we wrote a program and created a sitemap lets say abc.xml. Then we submitted the sitemap to google.

After 2 months, I can see that the google acknowledged the submission but never indexed these 8000+ pages.

561 indexed pages are from our blog post and pages.

I guess the mistakes we did was that we created the sitemap of 8000+ links and submitted when we should have done it in piecemeal basis. 2nd mistake I guess is that we do not update the abc.xml sitemap regularly which we thought of doing once all the pages were indexed.

What is suggested

a) to fix this problem

b) and how to get the pages indexed ?

Answer 1

To answer your question very specifically. Google prefers to index smaller sites and submitting a sitemap, while advisable for a site your size, may actually go largely ignored. Yes Google has read it. And yes Google is indexing pages, but I would surmise that Google is opting for indexing your site by following links more than the site map.

One site I submitted a sitemap for was fully indexed while the sitemap was hardly touched according to Google Webmaster Tools. This should not be a concern.

As well, any site can really take a while to get indexed especially any new site. Google is very good about discovering new sites and spidering them, however, any new site remains at the end of the fetch queue consistently up and until the site is well settled into the index and Google begins revisiting the site. Simply put, any new site will be at the end of the line for a while.

If your site is relatively new, it will likely take a while longer to get all of your pages in Google. You will see Google go in fits and spirts and it will frustrate the h311 out of you! Just be cool for a while. It will all work out for you just fine.

Answer 2

I'd personally just wait. Google can crawl a site without a sitemap - there is no reason for them to penalise a site for using one correctly.

My feeling is that it will be more to do with how they naturally index a new large site. They may have rules on how many pages they will index a day. Also: Wordpress generate a lot of pages of duplicate content - they may be ignoring some of these.

Have the indexed pages stopped going up altogether, or are they just going up slowly?

Google say...

Please note that submitting a Sitemap doesn't guarantee that all pages of your site will be crawled or included in our search results.

• https://support.google.com/webmasters/answer/34397?hl=en

As you start getting external links to your site, I'm sure the remainder of your site will be slowly indexed as they will see it as more important.

If there are any particularly vital pages that are not indexed, consider using the priority attribute in the sitemap, or try and get (natural) links pointing to it.