Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

(Note that I am going to use screenshots here because I suspect writing about this will change the behavior over time.)

If you do a Google search for

uiviewcontroller best practices

either with or without the quotes, you end up with results like this:

alt text

alt text

Note that none of these pages resolve to the actual Stack Overflow question containing those words in the title. They resolve to either a) sites that are mirroring our creative commons data and correctly pointing back to the source question without nofollow, as properly specified by our attribution requirements or b) our own internal links to the question, but not the actual question itself.

The actual page with the title ...

Custom UIView and UIViewController best practices?

... does exist at this URL ...

http://stackoverflow.com/questions/3300183/custom-uiview-and-uiviewcontroller-best-practices

... and apparently it is present in Google's index!

alt text

But why does it not appear when we search for

uiviewcontroller best practices

?

We know that

I don't get it. What are we doing wrong here?

share|improve this question
9  
@Jeff Atwood I've been noticing that other sites that mirror Stack Overflow content are coming up before Stack Overflow in searches for questions. I'm not sure what's causing it, but it doesn't give me a warm-fuzzy. –  George Stocker Nov 12 '10 at 22:00
    
Perhaps you haven't paid your Google bill this month. I wonder if it's content related...are there any stop words in say one of the ads that displayed when the page was indexed? –  Webjedi Nov 12 '10 at 22:06
1  
@george it's the above issue, or the page isn't indexed at all. –  Jeff Atwood Nov 12 '10 at 22:07
1  
Perhaps there's an error in your sitemap, but I can't see, because Stackoverflow blocks sitemap access to non-search-engines. Could you add the relevant snippet from your sitemap to the question? –  Dan Fabulich Nov 12 '10 at 22:07
    
@dan sitemap is irrelevant here; see webmasters.stackexchange.com/questions/4803/the-sitemap-paradox -- but you can assume this WAS in the sitemap of the last 50k active questions at the time it was asked. –  Jeff Atwood Nov 12 '10 at 22:21

5 Answers 5

There's not much you can do about this. There's a teensy bit you can do to help, but the problem is endemic to Google's index.

  • There isn't just one Google "index;" it's sharded. Pages with low PageRank appear in very few shards. Using site:stackoverflow.com forces your query onto a shard that has a larger fraction of your URLs indexed. That explains the contradictory results you see: the search results are coming from different shards.
  • PageRank applies to pages, not domains. Yes, really. Since the StackOverflow home page has high PageRank, the home page can flow PageRank to other question pages, but if your home page doesn't link to this particular Question page, then the Question's effective PageRank is very low.
  • You can also flow PageRank via your XML sitemap. According to this paper:

    The root of the domain is assumed to contain an implicit link to the Sitemaps file. In addition, the Sitemaps file is assumed to contain links to all of its URLs, either uniformly weighted or weighted by priority if this field is provided. Over this augmented graph, PageRank can be calculated for every URL in the Sitemaps page.

  • StackOverflow doesn't link to every question on the site; it only links to a tiny minority of questions for a very brief period of time. Furthermore, it's my understanding that you guys even delete links from your sitemap, only linking to the most recent questions. So many pages on the site receive no benefit from the home page's very high PageRank.

So, you can help this problem a little bit by making sure your XML sitemap is 100% complete (not just the newest questions, but ALL of them) and by making sure that every question page can be reached by a short chain of links starting from the home page.

But even that won't always work; you can't reasonably expect every page on your site to have high PageRank. In those cases, those pages will be crawled, but probably won't appear on a lot of shards, so Google won't always return them.

Good luck!

EDIT: Jeff hates sitemaps, so I updated this answer to make it easier for him to read.

share|improve this answer
1  
This is another good reason to have a site map thus answering webmasters.stackexchange.com/questions/4803/the-sitemap-paradox –  Adam Nov 12 '10 at 22:40
1  
I could not disagree more w/r/t Sitemaps; none of the empirical data we have bears this out. If Google can't spider your site, sitemap is worthless. However, Google should be able to spider questions?page=1 through questions?page=9999 and until about 2 weeks ago this was explicitly suppressed to them in favor of the sitemap (which, uh, didn't work). –  Jeff Atwood Nov 12 '10 at 22:57
1  
@dan even for sites which fit in the sitemap (eg they have less than 50k questions) we have seen this phenomenon of no weight given to the links. You're giving dangerous and incorrect -- in my experience -- advice w/r/t the sitemap. On top of that, you can clearly see IN THE SCREENSHOTS I POSTED that we link to this question, on multiple pages! Take a closer look... i.imgur.com/PfyMz.png –  Jeff Atwood Nov 12 '10 at 23:10
2  
@Jeff Go ahead and ignore everything I said about sitemaps. I think that's a mistake, but it's a small mistake; little harm done. Instead, please update your post with the chain of links from your home page to this Question page. Hint: if it goes through questions?page=5 and then ?page=10 and then ?page=15, it's probably not a very short chain; that's a problem for you. –  Dan Fabulich Nov 12 '10 at 23:40
2  
@Dan I suspect you're on to it, if the homepage of the site doesn't link to the page very directly, then you're telling Google that page is not important. If that page is important, then get linking to it :) For sites like this we really need a tool to answer the question: how far away from the homepage is a given page, and what does the site's linkgraph look like. –  Alex Black Nov 13 '10 at 0:38

I don't know if it makes a difference, but looking at efreedom's page source code I noticed they're using google_ad_section_start and google_ad_section_end markers (http://www.google.com/support/adsense/bin/answer.py?hl=en&answer=23168). Could this allow Google to better weigh the actual content of the site?

Also, the actual content seems to be closer to the top of the page, whereas with SO there's a lot of ceremony going on before the actual content (this may not matter either, just a guess).

share|improve this answer

Very short and super easy answer. Ping the question.

I work in SEO, and this is how I make my tens of dollars a year ;)... things showing up in the big G.

I have spent thousands upon thousands of dollars working on tool that speed all of this up for me.

2 things make you show up before the rest (for exact phrases)

  1. Spider Crawls
  2. Inbound links

Ping the page, ping the sitemap, and let the mirrors point back to you... There is a phrase we have in the SEO world, it is the Google Dance, you will bounce up and down the results latter while Google finds where to place you "permanently."

EDIT

For better clarity, here is a PDF about the different google bots http://www.telezent.com/telezent/Resources/FAMILY-OF-GOOGLE-CRAWLERS.pdf

share|improve this answer
2  
what does ping'ing a page mean? –  Peter Mortensen Dec 11 '10 at 13:52

An important SEO consideration is your site's internal linking scheme--internal linking affects both spiderability and the weighting of pages in your site.

One of the guidelines for internal linking is that pages should contain no more than ~100 links--more than 100 starts to look spammy (to a crawler) and dilutes the value of each link to the point where each link isn't very valuable at all, even though you're linking from a high-authority page.

It's interesting to note that the Stack Overflow home page has approximately 700 links in its source code, whereas Efreedom only has about 35. Individual topic pages on SO have 100+ links, while Efreedom only has about 30.

All that being said, the Google Index serving me results seems to be getting things right, with SO on top (at least for the 10 searches I tried).

share|improve this answer

I wonder if there's any location-specific issue here?

From my desk in central London (the UK version) I just Googled uiviewcontroller best practices and the first three results were SO, with a single efreedom link at #4. That's a google.co.uk search.

In general, I think I see SO links higher in Google's search results - I actually wasn't aware of eFreedom at all prior to today.

I note that Alexa's eFreedom entry has it at #568 for India - perhaps there's some particular bias with Indian-based searches?

share|improve this answer
    
I'd suggest google has updated its index several times since the question was asked almost a month ago. –  MrG Dec 9 '10 at 9:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.