We’re rewarding the question askers & reputations are being recalculated! Read more.
9

Nothing wrong with that, as Google confirmed, they crawl based on URLs' popularity so, in your case, that article must have more internal/external links than the others. URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index. Also, John Muller confirmed that on Twitter. In addition, how Google's ...


5

I think this help document from Google should be solving my problem: Change the crawl rate: On the Webmaster Tools Home page, click the site you want. Click the gear icon , then click Site Settings. In the Crawl rate section, select the option you want. The new crawl rate will be valid for 90 days.


3

It is my understand that all google bot crawler IP addresses can be identified by the RDNS which is set to something along the lines of "crawl-(IP ADDRESS).googlebot.com". The useragent you mentioned is also not used by the googlebot. Google lists the useragent strings of all their bots here. The IP address you posted is part of the netblock used by ...


3

You should also specify a crawl delay in robots.txt for all the other search engines (Yandex and Baidu can be quite aggressive in their crawling). Add this: User-agent: * Crawl-delay: 5 The crawl delay is in seconds. Make sure not to go too high - 5-10 seconds max should lighten the server load considerably. If you have 1000 new pages per day you want ...


3

You can't. XML sitemaps are for you tell search engines what pages you would like to have indexed. But it doesn't guarantee search engines will crawl or index those pages. In fact, there is no way to force search engines to index anything. They have their own criteria for what gets indexed and when and we cannot change that.


3

If you think that yandex is not obeying the Crawl-delay directive try signing up for Yandex Webmasters and validate your domain. Once the validation is done you can manually set the crawl rate under Indexing -> Crawl rate Consider reading this article on Yandex Webmasters Guide to Yandex Webmaster Tools


3

There is no need to worry about this. In fact, it's a good thing, this way google would be able to index your updated content and would increase the chances of your page ranking higher. If you, however, do not want to get your page indexed by google(for any reason whatsoever), there are multiple ways to do that. You can add a noindex tag on the head of ...


2

Pages that are not frequently updated or are not important in the eyes of a search engine, after time, will be crawled (visited) by search engines less and could eventually be dropped from Google's index altogether eventually. This doesn't mean there is an issue at all and is quite common for large sites with many pages.


2

You can use the "report a problem with Googlebot" link in Webmaster Tools to let the Googlebot team know about your crawling preferences. You can find it in the site's dashboard, in the gear-icon (top right), under "Site settings", "Crawl rate", "Learn more." They may sometimes be able to tweak things, or it might make sense to just keep it on automatic.


2

ANY crawler can identify itself as "Googlebot" or other search engine bot via its user agent. So the reason could be that some crawlers are "spamming" your site. I'd still advise that you to set a crawl delay in your robots.txt for all bots.


2

According to the Google Crawl Rate is determined by their algorithms. In order our website to be crawled very often we need to update the content of website frequently. Google has Caffeine Update on June 8, 2010. This mainly deals with providing users with fresh results by crawling the websites with fresh content. Means they crawl the fresh content often ...


2

Is it a significant strain on the host? It depends on what you would consider a significant drain on the host and the extent to which you are crawling data and the frequency with which you do it. Search engine spiders crawl site content very frequently and as long as they are doing it in a safe manner and following industry best practices on limiting the ...


2

Google uses an algorithm to decide how frequently to recrawl pages. While the exact mechanics are kept confidential the basics are that it is based on how large the site is, how high quality the site is, the number of other sites that link back to it, and how frequently the content on the pages changes. The figures you have posted seem quite reasonable crawl ...


2

The first thing to remember is that what may seem like a slow crawl rate may not necessarily be slow and that when it comes to crawl-rates concepts like slow become meaningless. My Site Hasn't Been Crawled Yet First things first, before Google can crawl a new site they need to know it exists. Google gets this information from a number of sources including.....


2

No, it won't. All you need to focus on is to make things easier for the end-user. Everything else doesn't matter anymore. Google's algorithms are very evolved and what's good for the user is good for robots and ultimately for rankings.


2

URL length does not have much impact on how Googlebot crawls. Changing your URLs will have the biggest effect. Googlebot will forever come back and crawl your old URLs. You will need to implement redirects to the new URLs and maintain those redirects indefinitely. If you change all your URLs, this can significantly increase the number of URLs that ...


1

How often Google indexes a site for new content is based on an algorithmic evaluation undertaken by Google. The same process is followed for both blogs as well as standard websites. The first step is that Google will undertake an initial crawl of your site. This can be from Google finding a link on another site that links back to your site, or from you as a ...


1

...blogs are on page 1-8....With google crawl being less likely to crawl over 3 clicks deep, will pages not immediately linked get penalized ... It really comes down to user experience. If you lay out your site as if it was a gigantic story book where users will have to click "next page" all the time, then Google (or any other search engine) might not have ...


1

Interesting question! To me it really seems like a matter of crawl budget vs. link juice. Is it perhaps possible to block certain parts of the result pages set? For instance those with particular parameters that nobody links to? Does it seem like you're already having an issue with the crawl budget on your site? If not, then keeping the link juice might be ...


1

http://tools.seobook.com/robots-txt/ Google does not support the crawl delay command directly, but you can lower your crawl priority inside Google Webmaster Central.


1

As I understand you have many pages that are less important that shouldn't be crawled (that often). I don't know if it will help removing the pages from the sitemap. The sitemap is not a selection of pages that will be crawled. When a search engine finds links outside the sitemap, they will just as well be crawled. Did you consider assigning a lower ...


1

May be Google check your robots.txt frequently, because you have moved your site content to new location. There is no any way to control robots.txt crawling rate. But, If your robots.txt did not blocked anything like any directory, then don't use robots.txt at all. For blocking other bots, you can use httacess. I have seen most of sites, don't use robots....


1

Googlebot crawls the web on a continuous basis. It also doesn't crawl your entire site at once. It may crawl some pages one day, some page the next, and some pages next week. Googlebot doesn't schedule crawls of site, rather it schedules crawls of individual pages. It may crawl your home page every day, but your deep pages every month. The rate at ...


1

You may want to consider adding a change-frequency tag to your XML sitemap: http://www.sitemaps.org/protocol.html Another couple things to look into would be ensuring last-modified headers match the creation (or modification) date of the user's account and, if it works for your application, cache related headers: http://www.mobify.com/blog/beginners-guide-...


1

If you haven't already done so, register an account with Google Webmaster Tools and add your domain to it then access your domain, and select the gear icon and visit Site Settings then you can select "Limit Google's maximum crawl rate" and select a value that you're OK with. Probably the lowest value works well for you which may be 0.002 requests a second (a ...


1

Vincent_Bing says: The graph is always plotted against your own timezone (as in the time zone on your system). To put this to the test, you can change simply change the timezone on your system and refresh the page - you will see that the pattern will change accordingly.


1

There is no standard. You will have to consider how robust the site is. Multiple servers and lots of bandwidth would be less taxed than a single server with less bandwidth. Also consider that some are paying for the bandwidth they use rather than having a set amount rated on a per second basis (like we are used to with DSL and cable). This would mean that ...


1

I dont have a definitive resource as an answer but from my own personal experience this happens if you tell google to crawl your pages manually by say fetching as google or submitting all of your pages to index "including linked pages" so if you have 300 pages and several of those pages have links to all 300 google might crawl all of them a couple times over....


1

Google expects refresh activity on a website, indicating it's live and have taken into account the fact that pages do get updated. You might want to permanent redirect important pages that have moved. If we believed such nonsense, our company website would never be updated. Google is continuously scanning from one end to the other. Get cracking and keep it ...


1

There is some logic to what I think you are saying, though mostly it really does not matter. I will give you some examples from my experience. Yes. I have been guilty of allowing Google to index the bulk of a large update to a site before moving things around again. But I do not stop working. I create back links, new content, update content and so on. When ...


Only top voted, non community-wiki answers of a minimum length are eligible