2

Sometimes a page on our site will show up in the Google SERPs that was previously not indexed (yet) due to a news story or blog posting on another site that points to it.

From what I understand, the Google FreshBot has crawled it. The FreshBot's job is to find information that is current or newsworthy to the time and get it indexed and into the SERPs quickly.

However, the pages that benefit from that drop out of the SERPs after two or three days. As a matter of fact, if I do an advanced query and request that page specifically from our site (using "site:"), it appears to not even be indexed any more.

What's happening to our Google FreshBot crawled pages?

Do pages that are indexed by the FreshBot still need to be re-indexed by the DeepBot?

For what it's worth, we're ordinarily getting indexed daily at a healthy rate, but we don't yet have all of our pages indexed (large site). It has been encouraging to see the effect of the FreshBot indexing, but watching it fade away is hard to understand. On our pages indexed as part of our normal crawls, our SERPs are strong -- without the added relevancy of news or other off-page links.

Improve this question
1
  • Note: I deleted an old question I had posted similar to this. I was able to gain some understanding, and I felt like a new question with more specifics would serve us better. My previous question had no comments and no answers.
    – Chris Adragna
    Dec 10, 2010 at 20:03

1 Answer 1

Reset to default
2

The conversation about "Freshbot" and "Deepbot" are pretty deprecated. I haven't heard those words used since the early 2000s on intermittent Geocities websites. And I can't find any good sources on them that aren't websites riddled with Adsense. The woes of Google indexing is such a nebulous mechanism that you're better off not trying to put such a fine point on the question by trying to peg distinct Google crawler specificities that probably don't even exist anymore.

More contemporary buzzword guesses for how Google indexing works for new websites include the Sandbox effect, but it's a better perspective to ask "why am I not getting indexed?", and then understanding that unless you have a reason to be penalized by Google or if Google simply doesn't understand your navigation heirarchy, you're likely just in the waiting game.

Sounds like your website is just new. Sometimes, new websites get fully indexed in less than a week. Sometimes, they finally get indexed after four months of wait no matter what you do.

Improve this answer
2
  • Thanks, Dobry, for the insight. We're getting indexed quite well, but with a large site, many pages are still not crawled (but we get visited daily at an increasing rate of pages per day). Still, some kind of quality exists so that the pages which get some news or social media buzz get included in SERPS, but then disappear after a few days. Pages that get crawled more naturally (without the effect of buzz) stick in the SERPS without falling out. I'm trying to better understand it all. Thanks!
    – Chris Adragna
    Dec 20, 2010 at 17:42
  • Dobry, I'm marking your response as the answer to my fairly specific question on Freshbot/Deepbot. I must add that we do see a difference in the performance in the SERPS based on the avenue to indexing. It's something I'm still very curious about.
    – Chris Adragna
    Dec 21, 2010 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.