5

Consider a website with a large number of pages. New pages are published regularly.

When publishing a new page the website operator wants to get the newly created paged crawled/indexed in Google as soon as possible. The website operator wants to minimize the time spent between publication and crawling/indexing.

Consider the site http://www.example.com/ with hundreds of thousands of pages.

The page page http://www.example.com/something/important-page.html is created at say 12:00. I want to get important-page.html crawled/indexed as soon as possible after 12:00. Ideally within seconds or minutes.

  • What options are available to try to get Google to crawl/index a specific newly created page as soon as possible?
Improve this question

5 Answers 5

Reset to default
4

Here are some steps you can take...

  1. Ping the page to indicate new or updated content. This automatically happens with some CMS systems like WordPress but others may need to run an external script. You only need to do this once for each new content item or for each update. Excessive automated pinging can cause your site to be ignored but don't get paranoid about a few simple manual updates here and there like some do who're being influenced by someone trying to sell a product.

  2. Twitter the link. Post the link on a public Facebook wall. Post the link on a related YouTube video. Similar activity with Google+ may help too but I haven't tested this.

  3. Post the link on other sites, particularly one with authority that has a high crawl rate. Don't be spammy with it though, make it relevant and worthwhile or don't place the link.

  4. Link to the new content from your home page or from another strong page on your site.

Improve this answer
2

What you need is a sitemap. You can find details about what a sitemap looks like at http://www.sitemaps.org/. There are various tools available for creating them automatically from your existing site.

However they are only a hint to search engines on how often pages are updated, and do not guarantee that they will index new pages as often as you say they are created or updated.

Improve this answer
2
  • Hi Paul. Sitemaps are great to increase crawlability of the entire site, but to get indexing for a specific page down to the sub-minute region I think you need additional and heavier artillery? :-)
    – knorv
    Nov 22, 2011 at 11:31
  • If the content is regular news/blog style content then having an RSS feed is usually more beneficial than a basic sitemap, especially if you have many subscribers.
    – DisgruntledGoat
    Nov 22, 2011 at 20:36
2

As with anything pertaining to Google, getting links to the new pages will help them find and index pages. The rate at which this happens is very much affected by the strength of those links. StackExchange sites, and other large sites like CNN.com, get crawled quickly thanks to having strong links pointing to those pages (internal links do count).

Improve this answer
7
  • Assume that the site as a whole has a lot of incoming links -- now, how can one speed up the indexation of a specific newly created page?
    – knorv
    Nov 22, 2011 at 13:27
  • You can't. Google will crawl your site whenever it feels like. You can't directly influence or control it.
    – John Conde ♦
    Nov 22, 2011 at 14:14
  • There are official and recommended ways to notify Google that newly created content has been created. Google may or may not use those signals as inputs to crawling.
    – knorv
    Nov 22, 2011 at 15:34
  • Telling Google about a page and when Google decides to crawl a page are two separate things.
    – John Conde ♦
    Nov 22, 2011 at 15:36
  • That's true. Sorry about not being clear about that distinction. What I'm talking about is making sure Google knows about the pages existance as soon as possible.
    – knorv
    Nov 22, 2011 at 15:55
1

Dynamically updated XML sitemaps submitted through Google Webmaster tools will ensure the content is indexed within a few days.

You could also try 'Fetch as Googlebot' within Google Webmaster tools as this contains a 'submit page to the index' option.

You could also try increasing your crawl rate under webmaster tools if your hosting is slow as this will encourage Googlebot to come back and re-index your site.

If your site has a good reputation and is updated frequently then you will have no problem getting it indexed otherwise after the above three (quicker) options have been tried...you need link building.

Improve this answer
0

What you need is to get google trust, then it will work as you expect by default.

One site I've to deal with has 1.5 million indexed pages, PR4 last time i check and a few days ago they run an experiment to check precisely how much time was taking google to publish new content. It turned to be between 5 and 10 minutes.

So, I would focus on get google confidence and trust in your content and authority, if you get that and you produce a lot of content, googlebot will be checking a lot your most changed pages and following the links on them.

That website only actions on publish were:

  • Link on homepage to new content.
  • Sitemap update & push to google.
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.