How long for Google to index all site maps?

Question

We recently uploaded new sitemaps, as the old ones were out of date and we've also upgraded our forum which changed the URLs of a lot of our website (we're 301 redirecting of course).

We uploaded the site map a few days ago. Initially the indexing seemed to be going well, but it paused on the numbers below here and appears to have been stuck here for over 24 hours:

Is this normal? Unfortunately, the users sitemap is probably the least important one but appears to be the bulk of the indexes so far.

Should we give it a bit more time, or perhaps make the links we consider more important a higher than 0.5 priority?

You can view our sitemaps here: https://www.scirra.com/robots.txt

Answer 1

The sitemaps with errors, I guess those speak for themselfs.

To answer the remainder of your question: Yes it is normal that they dont index all. This number will slowly increase in time, but cán stop before it'll reach 100%. That means they find some pages not worth their while.
That could be because of duplicate content, bad content/title/description, etc.

The priority in the sitemap has NO INFLUENCE on this. Otherwise everybody would just set it to 1.0 and get it over with. That is more for the users and other parsers. They might even give you more credit for giving pages the proper amount of priority rather than all to the max.

"has been stuck here for over 24 hours"

24 hours is nothing in SEO terms, thinking more in the range of 'weeks' as smallest units would be more accurate.

Answer 2

I had a look at your XMLs as well as some sections of the site, to answer your questions: there is no fixed time for the sitemaps to get indexed, it depends on a whole bunch of factors like size of your site, crawl friendliness of your site (more on this later), interna/external links to the site etc.

Now it actually does not matter if you list all URLs in the sitemap, if the crawler cannot find your links then they will not get indexed - even if they were in the sitemap.

For example: the link https://www.scirra.com/forum/ i.e. the parent node of your forums section is listed in the sitemap, if you check Google's cached version of this page you'll notice that it does not pick up any sub-forums links at all (run a search with cache:https://www.scirra.com/forum/ to see what I mean) So what I think is happening is that when Google crawler visits your forum home it does not find any of the links you placed in the sitemap (perhaps you render a different version based on user agent?) and when it sees these links in the sitemap it infers that these links are not important since they are not linked from any high importance page within the website. I randomly tried checking the cached version of some your sub-forums and sure enough, Google has not indexed them - probably because the crawler never saw them.

My other recommendation would be around structuring the sitemaps itself - why don't you use a sitemap index file (see this for more detail: https://support.google.com/webmasters/answer/71453?hl=en) rather than creating 8-10 different files in the robots file? Functionally it won't make a massive difference but it's a more elegant way of preparing XML files for larger sites. Sorry for that wall of text - hope it was helpful!!

Answer 3

To answer your question "how long", it depends on your site size, quality, consistency, and of course, popularity. Of course Google does what it wants too, so it really just depends on how it may feel that day.

Case 1) Popular forum with 1000+ avg 30 min sessions, above PR 5 --> Sitemaps are crawled and indexed multiple times a day, sometimes multiple times an hour. Its not uncommon for your posts/users on these forums to show in Google results within 10 minutes.

Case 2) Content rich forum with relevant targeted data, spam free, above PR 2 --> Sitemaps are crawled multiple times a day, up to once an hour, however its up to Google whether it indexes them. On a certain small fry forum we work with, ive seen posts appear in Google results in under 4 hours.

Case 3) Other small, off the radar forum, with diluted content, spam target, or too much competition --> Sitemaps are crawled a couple times a day and if it doesnt see changes happening, or no users are querying Google for these SERPS, it may take up to a week or more to index.

Note) Even if your sitemap errors are repaired, RDF is a different animal - unless you are a super popular site, it can take up to 2 months to fully index the widgets. Its even longer to forget if there is error - errors from RDF may stick for 8+ months (although often disabled in index immediately). If you fix your sitemap errors, and things still aren't being crawled right, look for any RDF error in webmaster tools > search appearance > structured data

Hope that helps!

Answer 4

First of all, it's recommended to use sitemap.xml name for the sitemap.

Moreover, Google takes time to index your website. If your website is down at that time it shows errors. So, you should just wait, and let Google index the website. For more popularity you can index your site in Bing, Yahoo, etc.