Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Is there some general description telling how Google crawls your website? I mean, not the frequency, but which URL is it going to explore next?

share|improve this question
    
Hmm I found some blog post on the topic: stateofsearch.com/… –  arivero Jul 9 '13 at 16:19
    
That's what I answered :) –  Mr. Alien Jul 9 '13 at 18:16
add comment

2 Answers

No. They have their own way of determining what pages get crawled and how often and it is not public information. Pages that are updated often and rank well seem to be crawled more often than static and poor ranking pages. But there is no way to predict when a page will be crawled or how often.

share|improve this answer
add comment

Search engines never reveal the way their algorithms work or people will just start exploiting them... Google also stopped respecting the meta description tags as well as meta keywords tags as developers used irrelevant keywords to top their websites in search engines, also this will lead to inaccurate search results ...

Which is the next website a Google bot will go for?

Well, this process is never pre defined, Google never ques up the websites to be crawled, say for example bot visited a URL, on that page it will get 10 links, 10 leads to 100 and so on. So more the websites points the links to a single website, more often is that website visited by the bots and this way it ranks up the page...

From Google Support :

For most sites, Googlebot shouldn't access your site more than once every few seconds on average. However, due to network delays, it's possible that the rate will appear to be slightly higher over short periods. In general, Googlebot should download only one copy of each page at a time. If you see that Googlebot is downloading a page multiple times, it's probably because the crawler was stopped and restarted.

Googlebot was designed to be distributed on several machines to improve performance and scale as the web grows. Also, to cut down on bandwidth usage, we run many crawlers on machines located near the sites they're indexing in the network. Therefore, your logs may show visits from several machines at google.com, all with the user-agent Googlebot. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server's bandwidth.

Control Crawl Rate

share|improve this answer
    
So, do you think that pagerank is calculated during the crawling? –  arivero Jul 9 '13 at 16:20
    
@arivero Not during the crawling, but more the page receives traffic and more the page is linked on other websites –  Mr. Alien Jul 9 '13 at 16:36
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.