2

Is it possible to schedule Google crawlers' crawling at a time when my website it turned on?

I host my website with Google's cloud product: Google Compute. To save costs I am thinking of turning the compute instance off, and would like to do that without affecting SEO.

If some other site links to my site, how does turning instances off and on affect their SEO and mine?

|improve this question
  • 1
    What does a compute engine have to do with your web site? The compute engine is usually for solving knotty problems, but not for handling web requests. – Stephen Ostermiller♦ Sep 5 at 16:03
  • The compute engine is my server, I thought this was a common usage. I prefer it to the Google cloud apps platform because it can be customized – deolcarsolutions Sep 5 at 17:40
  • Sorry, I was more familiar with the Amazon terminology. Their "compute" products are meant for data crunching and they have other instances tuned for web hosting. – Stephen Ostermiller♦ Sep 5 at 19:01
  • I've updated your question to be a bit more generic. When you say "turn off your instance", that would shut your website down, right? As opposed to turning off half your instances when your website isn't as busy. – Stephen Ostermiller♦ Sep 5 at 19:01
  • You can't pull your website completely offline without affecting SEO. Although you can suggest Google crawling times, you can't guarantee they'll follow those times, and as you mentioned real live visitors may follow a link to your site while it's down and assume your site is gone. You also can't control every spider - Googlebot isn't the only one out there. :) It would be wiser to find a different way to host the site that meets your budget needs while keeping a reasonable amount of uptime. – WebElaine Sep 6 at 13:52
2

No, you can't schedule when Google (or any other major search engine) crawls in the way you're looking for.

Most sites do have down time, and issuing the appropriate 503 HTTP code will let search engine bots know to come back later. But if that happens for too long or too often, search engines will tend to assume the site is permanently gone or at least too unreliable to rank well.

There would also be indirect negative effects on SEO. For example by being only intermittently available, the site will likely appear unreliable and so attract fewer links.

You can configure crawl rate for some search engines (Google being one of them), but that's not quite the same thing. Bing allows you to set a preferred time of day for crawling but, again, that's not quite what you're after.

|improve this answer
  • 1
    I agree you can't schedule Googlebot, but you can schedule BingBot crawl time of day. – Stephen Ostermiller♦ Sep 7 at 10:14
  • Fair point, added. – GDav Sep 7 at 10:18
  • The thing about Google ranking is that people point out how it is smart etc, but this is a contradiction if it can't even keep into account factors that makes its cloud platform attractive. Why should a site seo be punished if it is like a 9 to 5 business so to speak? – deolcarsolutions Sep 7 at 16:51
  • It's not for me to answer for Google, but it's extremely unusual for a website to be available only during certain hours. As a result, the web hasn't developed the mechanisms to deal with it. In other words Google has no reliable way of knowing the difference at vast scale between sites which "close" for the day and ones which are almost always broken. The latter is by far the more common scenario and, for a business predicated on providing reliably useful search results, is a reasonable basis on which to exclude a site from its index. – GDav Sep 12 at 13:19
  • Even if you could schedule when your site is crawled, you can't schedule when people do their internet searches and find your site. Your site needs to be available 24/7 to satisfy users. Here is a somewhat related question about having regular website downtime: Say a customer is an observant Jew and wants his site to be offline on Shabat - SEO problem? – Stephen Ostermiller♦ Sep 13 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.