Page appears indexed in Google but not findable for any search terms?

Question

(Note that I am going to use screenshots here because I suspect writing about this will change the behavior over time.)

If you do a Google search for

uiviewcontroller best practices

either with or without the quotes, you end up with results like this:

Note that none of these pages resolve to the actual Stack Overflow question containing those words in the title. They resolve to either a) sites that are mirroring our creative commons data and correctly pointing back to the source question without nofollow, as properly specified by our attribution requirements or b) our own internal links to the question, but not the actual question itself.

The actual page with the title ...

Custom UIView and UIViewController best practices?

... does exist at this URL ...

http://stackoverflow.com/questions/3300183/custom-uiview-and-uiviewcontroller-best-practices

... and apparently it is present in Google's index!

But why does it not appear when we search for

uiviewcontroller best practices

?

We know that

• Google contains this page in its index

• Our search terms match the title of the question

• Stack Overflow has much higher pagerank than the other sites that are mirroring this question under Creative Commons

I don't get it. What are we doing wrong here?

Answer 1

There's not much you can do about this. There's a teensy bit you can do to help, but the problem is endemic to Google's index.

• There isn't just one Google "index;" it's sharded. Pages with low PageRank appear in very few shards. Using site:stackoverflow.com forces your query onto a shard that has a larger fraction of your URLs indexed. That explains the contradictory results you see: the search results are coming from different shards.
• PageRank applies to pages, not domains. Yes, really. Since the StackOverflow home page has high PageRank, the home page can flow PageRank to other question pages, but if your home page doesn't link to this particular Question page, then the Question's effective PageRank is very low.

• You can also flow PageRank via your XML sitemap. According to this paper:

  The root of the domain is assumed to contain an implicit link to the Sitemaps file. In addition, the Sitemaps file is assumed to contain links to all of its URLs, either uniformly weighted or weighted by priority if this field is provided. Over this augmented graph, PageRank can be calculated for every URL in the Sitemaps page.

• StackOverflow doesn't link to every question on the site; it only links to a tiny minority of questions for a very brief period of time. Furthermore, it's my understanding that you guys even delete links from your sitemap, only linking to the most recent questions. So many pages on the site receive no benefit from the home page's very high PageRank.

So, you can help this problem a little bit by making sure your XML sitemap is 100% complete (not just the newest questions, but ALL of them) and by making sure that every question page can be reached by a short chain of links starting from the home page.

But even that won't always work; you can't reasonably expect every page on your site to have high PageRank. In those cases, those pages will be crawled, but probably won't appear on a lot of shards, so Google won't always return them.

Good luck!

EDIT: Jeff hates sitemaps, so I updated this answer to make it easier for him to read.

Answer 2

I don't know if it makes a difference, but looking at efreedom's page source code I noticed they're using google_ad_section_start and google_ad_section_end markers (http://www.google.com/support/adsense/bin/answer.py?hl=en&answer=23168). Could this allow Google to better weigh the actual content of the site?

Also, the actual content seems to be closer to the top of the page, whereas with SO there's a lot of ceremony going on before the actual content (this may not matter either, just a guess).

Answer 3

Very short and super easy answer. Ping the question.

I work in SEO, and this is how I make my tens of dollars a year ;)... things showing up in the big G.

I have spent thousands upon thousands of dollars working on tool that speed all of this up for me.

2 things make you show up before the rest (for exact phrases)

1. Spider Crawls
2. Inbound links

Ping the page, ping the sitemap, and let the mirrors point back to you... There is a phrase we have in the SEO world, it is the Google Dance, you will bounce up and down the results latter while Google finds where to place you "permanently."

EDIT

For better clarity, here is a PDF about the different google bots http://www.telezent.com/telezent/Resources/FAMILY-OF-GOOGLE-CRAWLERS.pdf

Answer 4

An important SEO consideration is your site's internal linking scheme--internal linking affects both spiderability and the weighting of pages in your site.

One of the guidelines for internal linking is that pages should contain no more than ~100 links--more than 100 starts to look spammy (to a crawler) and dilutes the value of each link to the point where each link isn't very valuable at all, even though you're linking from a high-authority page.

It's interesting to note that the Stack Overflow home page has approximately 700 links in its source code, whereas Efreedom only has about 35. Individual topic pages on SO have 100+ links, while Efreedom only has about 30.

All that being said, the Google Index serving me results seems to be getting things right, with SO on top (at least for the 10 searches I tried).

Answer 5

I wonder if there's any location-specific issue here?

From my desk in central London (the UK version) I just Googled uiviewcontroller best practices and the first three results were SO, with a single efreedom link at #4. That's a google.co.uk search.

In general, I think I see SO links higher in Google's search results - I actually wasn't aware of eFreedom at all prior to today.

I note that Alexa's eFreedom entry has it at #568 for India - perhaps there's some particular bias with Indian-based searches?