Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We recently uploaded new sitemaps, as the old ones were out of date and we've also upgraded our forum which changed the URLs of a lot of our website (we're 301 redirecting of course).

We uploaded the site map a few days ago. Initially the indexing seemed to be going well, but it paused on the numbers below here and appears to have been stuck here for over 24 hours:

enter image description here

Is this normal? Unfortunately, the users sitemap is probably the least important one but appears to be the bulk of the indexes so far.

Should we give it a bit more time, or perhaps make the links we consider more important a higher than 0.5 priority?

You can view our sitemaps here: https://www.scirra.com/robots.txt

share|improve this question

4 Answers 4

The sitemaps with errors, I guess those speak for themselfs.

To answer the remainder of your question: Yes it is normal that they dont index all. This number will slowly increase in time, but cán stop before it'll reach 100%. That means they find some pages not worth their while.
That could be because of duplicate content, bad content/title/description, etc.

The priority in the sitemap has NO INFLUENCE on this. Otherwise everybody would just set it to 1.0 and get it over with. That is more for the users and other parsers. They might even give you more credit for giving pages the proper amount of priority rather than all to the max.

"has been stuck here for over 24 hours"

24 hours is nothing in SEO terms, thinking more in the range of 'weeks' as smallest units would be more accurate.

share|improve this answer
    
The sitemaps with errors, I guess those speak for themselfs. The error is that they are empty, they are just overflows for when the previous ones get near 50,000 urls. The priority in the sitemap has NO INFLUENCE on this. Otherwise everybody would just set it to 1.0 and get it over with I was mainly refering to making sure the URLs we consider more important are indexed first. –  Tom Gullen Apr 17 at 10:17
    
See, those did speak for themselfs :p About the priority, in that context I don't know, I can't give an answer based on research :) –  Martijn Apr 17 at 10:19

I had a look at your XMLs as well as some sections of the site, to answer your questions: there is no fixed time for the sitemaps to get indexed, it depends on a whole bunch of factors like size of your site, crawl friendliness of your site (more on this later), interna/external links to the site etc.

Now it actually does not matter if you list all URLs in the sitemap, if the crawler cannot find your links then they will not get indexed - even if they were in the sitemap.

For example: the link https://www.scirra.com/forum/ i.e. the parent node of your forums section is listed in the sitemap, if you check Google's cached version of this page you'll notice that it does not pick up any sub-forums links at all (run a search with cache:https://www.scirra.com/forum/ to see what I mean) So what I think is happening is that when Google crawler visits your forum home it does not find any of the links you placed in the sitemap (perhaps you render a different version based on user agent?) and when it sees these links in the sitemap it infers that these links are not important since they are not linked from any high importance page within the website. I randomly tried checking the cached version of some your sub-forums and sure enough, Google has not indexed them - probably because the crawler never saw them.

My other recommendation would be around structuring the sitemaps itself - why don't you use a sitemap index file (see this for more detail: https://support.google.com/webmasters/answer/71453?hl=en) rather than creating 8-10 different files in the robots file? Functionally it won't make a massive difference but it's a more elegant way of preparing XML files for larger sites. Sorry for that wall of text - hope it was helpful!!

share|improve this answer

To answer your question "how long", it depends on your site size, quality, consistency, and of course, popularity. Of course Google does what it wants too, so it really just depends on how it may feel that day.

Case 1) Popular forum with 1000+ avg 30 min sessions, above PR 5 --> Sitemaps are crawled and indexed multiple times a day, sometimes multiple times an hour. Its not uncommon for your posts/users on these forums to show in Google results within 10 minutes.

Case 2) Content rich forum with relevant targeted data, spam free, above PR 2 --> Sitemaps are crawled multiple times a day, up to once an hour, however its up to Google whether it indexes them. On a certain small fry forum we work with, ive seen posts appear in Google results in under 4 hours.

Case 3) Other small, off the radar forum, with diluted content, spam target, or too much competition --> Sitemaps are crawled a couple times a day and if it doesnt see changes happening, or no users are querying Google for these SERPS, it may take up to a week or more to index.

Note) Even if your sitemap errors are repaired, RDF is a different animal - unless you are a super popular site, it can take up to 2 months to fully index the widgets. Its even longer to forget if there is error - errors from RDF may stick for 8+ months (although often disabled in index immediately). If you fix your sitemap errors, and things still aren't being crawled right, look for any RDF error in webmaster tools > search appearance > structured data

Hope that helps!

share|improve this answer

First of all, it's recommended to use sitemap.xml name for the sitemap.

Moreover, Google takes time to index your website. If your website is down at that time it shows errors. So, you should just wait, and let Google index the website. For more popularity you can index your site in Bing, Yahoo, etc.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.