Is there some general description telling how Google crawls your website? I mean, not the frequency, but which URL is it going to explore next?
-
Hmm I found some blog post on the topic: stateofsearch.com/…– ariveroCommented Jul 9, 2013 at 16:19
-
That's what I answered :)– Mr. AlienCommented Jul 9, 2013 at 18:16
Add a comment
|
2 Answers
No. They have their own way of determining what pages get crawled and how often and it is not public information. Pages that are updated often and rank well seem to be crawled more often than static and poor ranking pages. But there is no way to predict when a page will be crawled or how often.
answered Jul 8, 2013 at 16:17
Search engines never reveal the way their algorithms work or people will just start exploiting them... Google also stopped respecting the meta description tags as well as meta keywords tags as developers used irrelevant keywords to top their websites in search engines, also this will lead to inaccurate search results ...
Which is the next website a Google bot will go for?
Well, this process is never pre defined, Google never ques up the websites to be crawled, say for example bot visited a URL, on that page it will get 10 links, 10 leads to 100 and so on. So more the websites points the links to a single website, more often is that website visited by the bots and this way it ranks up the page...
From Google Support :
For most sites, Googlebot shouldn't access your site more than once every few seconds on average. However, due to network delays, it's possible that the rate will appear to be slightly higher over short periods. In general, Googlebot should download only one copy of each page at a time. If you see that Googlebot is downloading a page multiple times, it's probably because the crawler was stopped and restarted.
Googlebot was designed to be distributed on several machines to improve performance and scale as the web grows. Also, to cut down on bandwidth usage, we run many crawlers on machines located near the sites they're indexing in the network. Therefore, your logs may show visits from several machines at google.com, all with the user-agent Googlebot. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server's bandwidth.
-
So, do you think that pagerank is calculated during the crawling?– ariveroCommented Jul 9, 2013 at 16:20
-
@arivero Not during the crawling, but more the page receives traffic and more the page is linked on other websites Commented Jul 9, 2013 at 16:36