2

I m thinking of making a blog website and noticed that google search results appear as per Google's indexing upon the crawled website. And this web crawling by google happens like once in 2 weeks to a few months.

But as I'm gonna update my blog every single day with 10 articles in various categories every day, how can I make my blog gets listed in google search results if Google doesn't crawl my site on change or update of content?

In the same scenario, how does Google lists Top News websites on the top of the page feed without crawling their site every day?

2
  • Is your blog focused on news? May 24 at 12:14
  • 1
    yes, it is focused on news
    – Harry
    May 25 at 7:57

1 Answer 1

1

Googlebot doesn't re-crawl sites. Googlebot recrawls pages. Every page on your website will get recrawled at a different rate based on two factors:

  1. Its PageRank.
  2. How often it is updated.

Googlobot will recrawl low reputation pages that are not frequently updated every few months. This is appropriate for pages about old news articles. It would be wasteful and unnecessary for Googlebot to have to recrawl such pages every day.

The home page of your site should be your highest PageRank page. It should have a link from every other page on your site and it usually attracts more external links than any other page on your site. For a news website, Googlebot will eventually start crawling it much more often. Googlebot recrawls the home pages of the national cable news websites every minute or so.

However, it will take a long time and a lot of effort to get your home page re-crawled nearly as often. If you are starting off with a new site Googlebot will recrawl it every couple months, just like any other low-reputation page on the internet. Once Googlebot sees that it is updated frequently, it will start re-crawling more frequently: once a week or even once a day. It takes thousands if not hundreds of thousands of external links into your site to get to the point where Googlebot re-crawls your home page every hour or every minute.

In the meantime there are several things that you can do to get Google to see your new content more quickly:

Create an XML sitemap

You can build and submit a sitemap to Google. When you submit a sitemap to Google, Googlebot will come and crawl all the URLs listed in the sitemap.

Google will learn that your sitemap gets updated frequently and start to fetch it more often. In the meantime you might want to submit the sitemap to Google daily as you add new content to your site.

Unfortunately, just because Googlebot crawls your URLs quickly, it doesn't mean that Google will choose to index your pages. In fact, just including a page in an XML sitemap without having links to it usually won't get Google to index the page. See The Sitemap Paradox.

Ping Google about new content

There is a way to ping Google about a new page the minute is created. Many content management systems like WordPress implements this. Using WordPress usually gets your new pages crawled very quickly. See How does wordpress submit new sites to search engines?

Like XML sitemaps, just because Googlebot comes to crawl the content quickly, it doesn't mean that the content will get indexed quickly. Once your site has enough reputation, Google will start to index new pages when it gets pinged. New questions on this site usually get indexed within minutes. However, for new websites with low reputation, Google just isn't willing to do that.

Sign up for Google News

For news websites specifically, you can ask to be included in Google News. That is what powers much of Google's news content. Google has instructions for doing so here.

Once your site is part of Google News, Googlebot will crawl your XML feed more frequently and your site will get listed along with other news websites.

Link to new content from every page

If every page on your site has a list called "Today's Headlines" that links to your most recent news stories, it will help the stories get found and indexed more quickly. Googlebot will crawl a few different pages from your site every day, even if any individual page on your site gets crawled every couple months. Then no matter which page Googlebot crawls today, it will see links to the most recent pages that you want Googlebot to crawl and include in the search index.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.