Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We're seeing some pages that exist in our sitemap.xml but are inexplicably missing from Google's public search index.

You can't download http://superuser.com/sitemap.xml -- we protect this file because there have been issues with it in the past -- but googlebot can. We have verified via Google Webmaster Tools that the sitemap.xml file was pulled down today and is rated OK with no errors (green checkmark).

alt text

The sitemap.xml contains a list of the last 50,000 questions on our site that were asked. For example, this question ...

http://superuser.com/questions/201610/how-to-see-the-end-of-a-long-chain-of-symbolic-links

... exists in the sitemap.xml as ...

<url>
<loc>http://superuser.com/questions/201610/how-to-see-the-end-of-a-long-chain-of-symbolic-links</loc>
<lastmod>2010-10-20</lastmod>
<changefreq>daily</changefreq>
<priority>0.2</priority>
</url>

Searching for "How to see the end of a long chain of symbolic links" gives only one result to questionhub.com which is scraping our data (a whole different problem).

You can increment the question count number and do an exact search for the question title and you will see this pattern persist.

These urls are in sitemap.xml but they are not showing up in Google's index -- and yet they show up on sites that scrape our creative commons data. Why would that be?

share|improve this question
5  
You could always ask in the google webmaster central forums. google.com/support/forum/p/Webmasters?hl=en –  Alex Black Oct 21 '10 at 19:41
    
Something is definitely wrong. THIS question is already indexed in Google, yet the linked question on superuser STILL does not show up in the index. –  Michael Pryor Oct 22 '10 at 14:11
    
Jeff might think about just asking Matt Cutts. I've seen them talking to each other a few times on Twitter. He's usually pretty willing to help. –  Virtuosi Media Oct 22 '10 at 16:57
3  
FWIW We're currently seeing some issues with indexing of new content on some sites. There's a thread in our Help Forums at google.com/support/forum/p/Webmasters/… about this. The URL that you mentioned appears to be affected. I imagine this will be resolved shortly, but have no fix timeframe available. Thanks for your patience. –  John Mueller Oct 22 '10 at 18:39
1  
It looks like this is resolved now :-). I tried a few of the new questions from the site and they were all indexed. Woot! –  John Mueller Oct 24 '10 at 10:40

6 Answers 6

up vote 9 down vote accepted

It looks like Google was having some technical crawl problems this week, that sound remarkably like what we were experiencing:

http://searchengineland.com/is-google-broken-sites-big-small-seeing-indexing-problems-53701

No one seems to be immune from a Google indexing problem that has many site owners baffled. Blogs and websites, big and small, aren’t being indexed as quickly as they normally are — if they’re being indexed at all.

...

John from Google replied to the thread in the Webmaster forums saying:

Just to be clear, the issues from this thread, which I have reviewed in detail, are not due to changes in our policies or changes in our algorithms; they are due to a technical issue on our side that will be visibly resolved as soon as possible (it may take up to a few days to be visible for all sites though)

share|improve this answer

Google doesn't make any offer or guarantee that pages in a sitemap will be indexed.

My experience has been that a page has to be linked-to (from a page of some authority) to show up. Is that page/question linked to directly/indirectly from a page with some authority?

E.g. if the superuser.com homepage (which presumably has many inlinks) linked directly to this question, or linked to it indirectly through a number of other pages, then you could expect it to be indexed.

From google:

Google doesn't guarantee that we'll crawl or index all of your URLs. However, we use the data in your Sitemap to learn about your site's structure, which will allow us to improve our crawler schedule and do a better job crawling your site in the future. In most cases, webmasters will benefit from Sitemap submission, and in no case will you be penalized for it.

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184

share|improve this answer
4  
Superuser should have sufficient links and PR to get those pages indexed with or without a sitemap. And minor pages get listed all the time. In fact they make up the majority of the index. I suspect something else is the culprit. –  John Conde Oct 21 '10 at 19:38
    
Agreed, the site has lots of PR and inlinks. But, is there a chance that the page in question doesn't have inlinks? If superuser.com (by chance) doesn't link to the page, then what does that tell google? it says the page is not important. –  Alex Black Oct 21 '10 at 19:39
2  
The page was definitely linked from the front page, and continues to be linked by from a number of other pages. SE sites are very cross-link heavy. –  Kevin Montrose Oct 21 '10 at 19:48
1  
at one point yesterday one of my hits for a test question was the superuser.com homepage -- with the target URL visible on it, even in the Google cache! And yet the question itself was not indexed. Very odd. –  Jeff Atwood Oct 21 '10 at 20:03
2  
absolutely -- click on the HOT tab on the home page, or the WEEKLY or MONTHLY tab. Right there.. –  Jeff Atwood Oct 21 '10 at 20:14

I think google might be having a hard time indexing your web pages, 50.000 is alot. So my suggestion would be breakdown your sitemap into pieces like so

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <sitemap>
      <loc>http://www.example.com/sitemap1.xml.gz</loc>
      <lastmod>2004-10-01T18:23:17+00:00</lastmod>
   </sitemap>
   <sitemap>
      <loc>http://www.example.com/sitemap2.xml.gz</loc>
      <lastmod>2005-01-01</lastmod>
   </sitemap>
</sitemapindex>

If you breakdown you will have a better luck of having those 50.000 urls indexed.

Sitemaps.org explanation of the issue

You can provide multiple Sitemap files, but each Sitemap file that you provide must have no more than 50,000 URLs and must be no larger than 10MB (10,485,760 bytes). If you would like, you may compress your Sitemap files using gzip to reduce your bandwidth requirement; however the sitemap file once uncompressed must be no larger than 10MB. If you want to list more than 50,000 URLs, you must create multiple Sitemap files.

If you do provide multiple Sitemaps, you should then list each Sitemap file in a Sitemap index file. Sitemap index files may not list more than 50,000 Sitemaps and must be no larger than 10MB (10,485,760 bytes) and can be compressed. You can have more than one Sitemap index file. The XML format of a Sitemap index file is very similar to the XML format of a Sitemap file.

http://sitemaps.org/protocol.php

share|improve this answer
2  
Sitemaps with 50,000 pages are very common. In fact someone recently posted a screenshot from their webmaster account showing Google having indexed almost all 50,000 of those pages. And I suspect superuser is more more popular (e.g. has better link popularity) than that other site. –  John Conde Oct 21 '10 at 19:57
1  
"You have more than 50,000 URLs to list. That's the maximum that one Sitemap can include." sitemaps.blogspot.com/2005/08/using-sitemap-index-files.html –  Jeff Atwood Oct 21 '10 at 20:07
1  
If you have a sitemap for each day those never change after the day is over, so that sitemap doesn't have to be accsessed again, they can crawl the links that they already indexed for changes, so google doesn't have to go trough 50,000 urls ever day to see which are old and which are new. –  Sevki Oct 21 '10 at 20:17
    
@sevki the oldest 50,001th question by ACTIVITY DATE (new answers, edits, etc bump this date) will not be in the sitemap. Bear in mind that superuser has only 55k questions in total. –  Jeff Atwood Oct 21 '10 at 20:56
    
@Jeff but SO.com has 1,014,782 and 964,782 isn't in a sitemap so google or bing doesn't know when they were last modified.. doesn't that increace your crawls. anyways I don't want to be annoying just trying to help out, I sent you an email with some more details. –  Sevki Oct 21 '10 at 21:06

It appears that Google is stating that 46,514 submitted links are in the index. Could it be an issue with (I hate to say it) but page ranking? The scraping sites may be doing a better job cross-linking etc and being ranked higher. Just a thought.

This search site:superuser.com How to see the end of a long chain of symbolic links also appears to be fetching your sitemap.xml correctly, albeit not returning the expected results.

share|improve this answer
    
That scraping site gives attribute to superuser.com as the original author (although they could be more explicit about it) so Google should know they are the content's original author and give them precedence over the scraping sites. –  John Conde Oct 21 '10 at 19:39
    
@john correct, we require attribution with follow, as documented on blog.stackoverflow.com/2010/08/defending-attribution-required –  Jeff Atwood Oct 21 '10 at 20:04
    
that cached sitemap is "as it appeared on 17 Oct 2010 05:40:35 GMT", 4 days ago as of the time I am writing this, so it doesn't have a lot. I spot checked a few URLs in that cached sitemap.xml and they do exist as question pages in google as well. –  Jeff Atwood Oct 21 '10 at 20:26
    
@john can you give an example of how they give the attribution. Thx –  Greg B Oct 21 '10 at 21:53
    
@Greg, Just look for the superuser logo –  John Conde Oct 22 '10 at 1:12

With this type of thing there are a lot of potential answers.

I'd start by asking how many pages you actually have. (you submitted 50,000 URLs a quick site:superuser.com show 125,000 indexed do you think you only have 50K URLs and are submitting all of them yet Google is finding 2-3 copies of each page? or maybe you have 1Mil URLs and only 12.5% are getting indexed) getting the big picture helps to direct where to look for issues.

If nothing seems wrong with step one, I'd move onto content, it looks like QH has a whole lot more content on their page and link out many other "resources" despite the fact that all their content is scraped it's possible Google considers their page more useful since they provide more resources/information to the user. If they are considered the authority and all your content is the same as theirs it's possible Google won't index yours even though you are the original.

If you're convinced that is not the issue build some high quality links to it, blog this question on some popular employee blogs or ask some friends to blog about it, perhaps if you have SEO friends that run popular blogs they'd write a case study about it etc.

If you get a lot of strong links and it's still not getting indexed look for reasons it might be penalized (in most cases this won't be the issue but it never hurts to check).

If none of this works then 9 times out of 10 it's a simple technical issue that's been overlooked (robots exclusion or something similar).

If you're still have no answer after going through this ask Google and hope they get you an answer.

share|improve this answer

The question was just asked yesterday - give googlebot a chance, you aren't the only site on the Internet that he has to crawl ya know :)

If questions are normally indexed within a day or so, and a week goes by and that one still isn't indexed, then I might be concerned. But certainly not after 1 day.

share|improve this answer
1  
They usually appear within an hour. So I agree, I should give it time, but relative to its usual frequency... I have. –  Michael Pryor Oct 21 '10 at 21:57
    
@michael make sure you're comparing apples to apples though -- Google seems to index stackoverflow.com at a MUCH higher rate than our other sites. –  Jeff Atwood Oct 22 '10 at 2:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.