Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Consider a website with a large number of pages. New pages are published regularly.

When publishing a new page the website operator wants to get the newly created paged crawled/indexed in Google as soon as possible. The website operator wants to minimize the time spent between publication and crawling/indexing.

Consider the site http://www.example.com/ with hundreds of thousands of pages.

The page page http://www.example.com/something/important-page.html is created at say 12:00. I want to get important-page.html crawled/indexed as soon as possible after 12:00. Ideally within seconds or minutes.

  • What options are available to try to get Google to crawl/index a specific newly created page as soon as possible?
share|improve this question
add comment

5 Answers

Here are some steps you can take...

  1. Ping the page to indicate new or updated content. This automatically happens with some CMS systems like WordPress but others may need to run an external script. You only need to do this once for each new content item or for each update. Excessive automated pinging can cause your site to be ignored but don't get paranoid about a few simple manual updates here and there like some do who're being influenced by someone trying to sell a product.

  2. Twitter the link. Post the link on a public Facebook wall. Post the link on a related YouTube video. Similar activity with Google+ may help too but I haven't tested this.

  3. Post the link on other sites, particularly one with authority that has a high crawl rate. Don't be spammy with it though, make it relevant and worthwhile or don't place the link.

  4. Link to the new content from your home page or from another strong page on your site.

share|improve this answer
add comment

What you need is a sitemap. You can find details about what a sitemap looks like at http://www.sitemaps.org/. There are various tools available for creating them automatically from your existing site.

However they are only a hint to search engines on how often pages are updated, and do not guarantee that they will index new pages as often as you say they are created or updated.

share|improve this answer
    
Hi Paul. Sitemaps are great to increase crawlability of the entire site, but to get indexing for a specific page down to the sub-minute region I think you need additional and heavier artillery? :-) –  knorv Nov 22 '11 at 11:31
    
If the content is regular news/blog style content then having an RSS feed is usually more beneficial than a basic sitemap, especially if you have many subscribers. –  DisgruntledGoat Nov 22 '11 at 20:36
add comment

As with anything pertaining to Google, getting links to the new pages will help them find and index pages. The rate at which this happens is very much affected by the strength of those links. StackExchange sites, and other large sites like CNN.com, get crawled quickly thanks to having strong links pointing to those pages (internal links do count).

share|improve this answer
    
Assume that the site as a whole has a lot of incoming links -- now, how can one speed up the indexation of a specific newly created page? –  knorv Nov 22 '11 at 13:27
    
You can't. Google will crawl your site whenever it feels like. You can't directly influence or control it. –  John Conde Nov 22 '11 at 14:14
    
There are official and recommended ways to notify Google that newly created content has been created. Google may or may not use those signals as inputs to crawling. –  knorv Nov 22 '11 at 15:34
    
Telling Google about a page and when Google decides to crawl a page are two separate things. –  John Conde Nov 22 '11 at 15:36
    
That's true. Sorry about not being clear about that distinction. What I'm talking about is making sure Google knows about the pages existance as soon as possible. –  knorv Nov 22 '11 at 15:55
show 2 more comments

Dynamically updated XML sitemaps submitted through Google Webmaster tools will ensure the content is indexed within a few days.

You could also try 'Fetch as Googlebot' within Google Webmaster tools as this contains a 'submit page to the index' option.

You could also try increasing your crawl rate under webmaster tools if your hosting is slow as this will encourage Googlebot to come back and re-index your site.

If your site has a good reputation and is updated frequently then you will have no problem getting it indexed otherwise after the above three (quicker) options have been tried...you need link building.

share|improve this answer
add comment

What you need is to get google trust, then it will work as you expect by default.

One site I've to deal with has 1.5 million indexed pages, PR4 last time i check and a few days ago they run an experiment to check precisely how much time was taking google to publish new content. It turned to be between 5 and 10 minutes.

So, I would focus on get google confidence and trust in your content and authority, if you get that and you produce a lot of content, googlebot will be checking a lot your most changed pages and following the links on them.

That website only actions on publish were:

  • Link on homepage to new content.
  • Sitemap update & push to google.
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.