7

I'm using Google CSE on my site. As soon as new entity is published, it's immediately added to sitemap.xml rendering script too. How soon after Googlebot fetches new sitemap new entities should be searchable with CSE?

I added a few entities last week. I checked web server logs and Googlebot last downloaded my sitemap on Apr 10th, at the same time it visited new entities pages. Today is Apr 12th and I still can't find my new items both in CSE and in regular google search like

"my entity name" site:mysite.com

Is this normal? How soon will I be able to find new pages? They may or may not have internal links from other site pages that are already indexed and periodically refreshed, but new items are always added to sitemap.

Improve this question
3
3

There is no fixed time from when Google crawls a site to when it is available on the index. A few years ago it would have taken anywhere up to 6 weeks for content to be added due to Google rebuilding the main index in batches, now with the live updating it is meant to take longer but it could be added to the index any time from a few days to a few weeks after the site has been crawled. It depends on a large number of factors, many of which Google won't release as it classes them proprietary.

Once you submit your URL, you might need to wait some time for Google to process your request as well as crawl and index the page. Also, understand that we can't guarantee that Google will index all your changes as Google relies on a complex algorithm to update indexed materials.
- From Google (https://support.google.com/webmasters/answer/6065812?hl=en)

Improve this answer
5
  • thing is the page was indexed, I checked web server logs and Googlebot requested new page. so it also takes some time for page to be included to search after it was indexed?
    – Oleg Shemetov
    Apr 12 '16 at 8:33
  • That's correct. You stated in your question that the sitemap and new pages where fetched on the 10th of this month, 2 days (depending on timezone) isn't long enough to see the changes reflected in the main Google index. As stated in the quote which I took directly from Google's help topic Google uses a complex algorithm to update the indexed materials, a full explanation of how the Google index operates is beyond the scope of this question but suffice to say it involves a huge amount more than just adding your pages to the index is also involves changes to many other index records.
    – Chris Rutherfurd
    Apr 12 '16 at 8:38
  • It really depends on the site, too. New questions on Stack Overflow generally show up on Google within mere minutes.
    – Lightness Races in Orbit
    Apr 12 '16 at 10:49
  • @LightnessRacesinOrbit - This is where the complex formula comes into play. Sites like Stack Overflow are detected by Google to have new content all the time and to be fairly good content to Google monitors sites like that more frequently and updates the index faster for them. Your average site may only be updated in the index every few weeks but popular high traffic and more importantly highly dynamic sites like Stack Overflow and other similar sites are indexed a hell of a lot faster.
    – Chris Rutherfurd
    Apr 12 '16 at 11:03
  • @PlanetScaleNetworks: Exactly.
    – Lightness Races in Orbit
    Apr 12 '16 at 11:35
1

If you want your pages indexed faster you need to do more than add them to your sitemap. A sitemap doesn't pass any link juice to the pages and Google usually doesn't index pages that have no link juice. See The Sitemap Paradox for more information.

You need to link new pages from other pages on your site. It might be worth having a "recently published" list on your home page for example. Your home page is a high PageRank page that can usually pass enough link juice to get the items linked there indexed.

Another way to let search engines know about your new content is to use ping services that alert them. The blogging software WordPress has this functionality built in and have a wiki page explaining how it is done. They rely mostly on Ping-O-Matic to spread the word to the various search engines about your updated content.

Improve this answer
1

You have to read this question carefully.

It is about the Google Custom Search Engine which is not the Google Search Engine. It is a feature whereby a site can have a custom search engine for the site. This, at least, is a site search feature.

These are two different things though there is some relationship between the two.

Assuming that a site owner opts for and implements the Google CSE feature and not all of the sites pages have been indexed into the Google Search Engine, what happens?

Google, for CSE customers, guarantees that all of a sites pages will be indexed quickly. This is providing that there is not a penalty for the site that gets in the way. Things have changed enough recently with all of the updates in 2015. It is more possible that a penalty will get in the way and therefore paramount that a site make sure that a penalty does not exist.

Google will completely refetch pages and fetch previously unknown pages it has not yet seen. It will use the sitemap as an audit to ensure that all of the pages are fetched. Google will attempt to do this very quickly though, depending upon the site size and speed, can still take weeks or months.

Why does Google do this?

To ensure that the CSE is complete. The Google Search Engine and Custom Search Engine work using the same index. If the site is not fully indexed, being a CSE customer will ensure that Google will, at least, attempt to index the entire site.

Once this process is complete, then what happens?

We go back to how Google behaves normally and much of that depends upon you.

Ask yourself:

  • How fresh is my site overall?
  • How fresh is any particular page?
  • It that page linked?
  • How important does the link to the page appear?
  • How big is my site?
  • How popular is my site?
  • How does my site rank with inbound links, social media engagement, citations, CTR (click-through rate) from the SERPs (search engine result page), etc.?

Assuming all is good, then Google will see new pages very quickly. If not, then it can take quite a while depending upon all of the above factors. In short, it can take minutes or weeks. On rare occasions, months.

Does Google's reading of my sitemap mean finding pages more quickly?

No. Not always. Google may chose to use your sitemap only to audit whether it can properly fetch your sites pages adequately. This is more common than not. Google is surprisingly old-school. It will prefer and continue to index your site as it has always done for all sites from the beginning of time if it can. Having a sitemap most of the time adds no benefit to the site short of pages behind a login or paywall or a site that is enormous.

So adding a page to your sitemap may do little to nothing. It is important that there be a link to any new page in a place of importance, such as, the home page.

But does that mean that pages that are not linked will not be found by Google?

No. If there are pages listed in a sitemap that are not linked, Google will fetch them and index them. However, Google does not like this. Google believes that all pages should be made available through the site itself. Google also believes that large numbers of pages found in the sitemap that are not linked is a signal for a spam site. They cannot be more wrong on this. For this reason, pages found following links on the site is indexed first and pages found in the sitemap second. This may not be strictly the case, but definitely close enough to be generally true.

You and I know that it is not possible to link a huge number of pages and Google also does not like navigational pages. As well, these pages can be cumbersome and bad for user experience (UX). I dropped mine a long time ago and intended to use the CSE as a valid replacement.

Does the CSE replace linking to a large quantity of pages?

Not very well. Which is a shame. It is the perfect opportunity for a webmaster to use the sitemap to signal to Google all of the pages within a site and use the CSE as a mechanism for a user to find content that cannot be effectively linked to.

Google seems to be the grumpy bear in this otherwise perfect process they created. It beats you up for having pages in your sitemap that is not linked on the site, it beats you up if you are frustrated and attempt to link these pages through navigational pages, and it takes it's sweet time indexing pages found in the sitemap and not found through links.

In addition, site search is not such a popular feature and seems to be under utilized in most sites. It is difficult to convert a user into a site search user. It is a sad fact. Most will go back to the SERPs and continue searching even if their next click is back on your site. Crazy huh?

So what are you to do?

  • Do everything you can to increase site search through conversion.
  • Make sure that all new pages are linked and signal importance even temporarily.
  • Find a replacement for navigational pages other than the CSE (not to replace CSE).
  • Keep your site fresh and popular and promote it on social media.
  • Let God and Google sort it out from there. In otherwords, do not worry beyond what you can control and do not try and control Google.
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.