10

Nothing wrong with that, as Google confirmed, they crawl based on URLs' popularity so, in your case, that article must have more internal/external links than the others. URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index. Also, John Muller confirmed that on Twitter. In addition, how Google's ...


5

I think this help document from Google should be solving my problem: Change the crawl rate: On the Webmaster Tools Home page, click the site you want. Click the gear icon , then click Site Settings. In the Crawl rate section, select the option you want. The new crawl rate will be valid for 90 days.


5

You need two things: Time External links Googlebot will eventually increase its crawl rate and crawl your entire site. Getting 2 million pages crawled could take a year or two. Even then Google may choose to index a small percentage of them. Getting 90% of your pages indexed could take five years. The best way to speed up Google's crawl rate is by ...


4

If you think that yandex is not obeying the Crawl-delay directive try signing up for Yandex Webmasters and validate your domain. Once the validation is done you can manually set the crawl rate under Indexing -> Crawl rate Consider reading this article on Yandex Webmasters Guide to Yandex Webmaster Tools


4

There is no need to worry about this. In fact, it's a good thing, this way google would be able to index your updated content and would increase the chances of your page ranking higher. If you, however, do not want to get your page indexed by google(for any reason whatsoever), there are multiple ways to do that. You can add a noindex tag on the head of ...


4

It is not a good idea to let Googlebot crawl URLs that can be constructed with combinations of parameters. When you allow your categories to be filtered, my advice is to only allow Googlebot to crawl pages with a single filter applied, even if you allow users to view multiple filters at the same time. To implement this, you need to change your URLs: /...


3

You should also specify a crawl delay in robots.txt for all the other search engines (Yandex and Baidu can be quite aggressive in their crawling). Add this: User-agent: * Crawl-delay: 5 The crawl delay is in seconds. Make sure not to go too high - 5-10 seconds max should lighten the server load considerably. If you have 1000 new pages per day you want ...


3

It is my understand that all google bot crawler IP addresses can be identified by the RDNS which is set to something along the lines of "crawl-(IP ADDRESS).googlebot.com". The useragent you mentioned is also not used by the googlebot. Google lists the useragent strings of all their bots here. The IP address you posted is part of the netblock used by ...


3

You can't. XML sitemaps are for you tell search engines what pages you would like to have indexed. But it doesn't guarantee search engines will crawl or index those pages. In fact, there is no way to force search engines to index anything. They have their own criteria for what gets indexed and when and we cannot change that.


2

According to the Google Crawl Rate is determined by their algorithms. In order our website to be crawled very often we need to update the content of website frequently. Google has Caffeine Update on June 8, 2010. This mainly deals with providing users with fresh results by crawling the websites with fresh content. Means they crawl the fresh content often ...


2

Pages that are not frequently updated or are not important in the eyes of a search engine, after time, will be crawled (visited) by search engines less and could eventually be dropped from Google's index altogether eventually. This doesn't mean there is an issue at all and is quite common for large sites with many pages.


2

You can use the "report a problem with Googlebot" link in Webmaster Tools to let the Googlebot team know about your crawling preferences. You can find it in the site's dashboard, in the gear-icon (top right), under "Site settings", "Crawl rate", "Learn more." They may sometimes be able to tweak things, or it might make sense to just keep it on automatic.


2

ANY crawler can identify itself as "Googlebot" or other search engine bot via its user agent. So the reason could be that some crawlers are "spamming" your site. I'd still advise that you to set a crawl delay in your robots.txt for all bots.


2

Is it a significant strain on the host? It depends on what you would consider a significant drain on the host and the extent to which you are crawling data and the frequency with which you do it. Search engine spiders crawl site content very frequently and as long as they are doing it in a safe manner and following industry best practices on limiting the ...


2

Google uses an algorithm to decide how frequently to recrawl pages. While the exact mechanics are kept confidential the basics are that it is based on how large the site is, how high quality the site is, the number of other sites that link back to it, and how frequently the content on the pages changes. The figures you have posted seem quite reasonable crawl ...


2

The first thing to remember is that what may seem like a slow crawl rate may not necessarily be slow and that when it comes to crawl-rates concepts like slow become meaningless. My Site Hasn't Been Crawled Yet First things first, before Google can crawl a new site they need to know it exists. Google gets this information from a number of sources including.....


2

From Yandex Webmastertools Support From February 22, 2018, Yandex doesn't take into account the Crawl-delay directive. So what you need to do is to login to Yandex.Webmasters and set speed: To set the speed at which robots downloads the site pages, use the Site crawl rate in Yandex.Webmaster.


2

No, it won't. All you need to focus on is to make things easier for the end-user. Everything else doesn't matter anymore. Google's algorithms are very evolved and what's good for the user is good for robots and ultimately for rankings.


2

URL length does not have much impact on how Googlebot crawls. Changing your URLs will have the biggest effect. Googlebot will forever come back and crawl your old URLs. You will need to implement redirects to the new URLs and maintain those redirects indefinitely. If you change all your URLs, this can significantly increase the number of URLs that ...


2

Have you looked at limiting Google's bot threshold? Per Google, If Google is making too many requests per second to your site and slowing down your server, you can limit how fast Google crawls your site. The one caveat is you cannot limit to a specific folder/directory. Per Google: You cannot change the crawl rate for sites that are not at the ...


1

Short answer: you can't force Google to crawl your site any faster. Why not add 301 redirects to those discontinued products to something comparable instead? Temporary promotional text on title tags is not likely a good SEO approach when the crawl rate can't be prioritized to keep Google's indexes of your site up to date in real-time. If you must control ...


1

It varies, depending mostly on how many links are pointing to your site. If you have a lot of inbound links Google will come crawl faster - if not, it may take weeks. It helps to make sure you have the site verified in Search Console and resubmit your XML sitemap(s).


1

...blogs are on page 1-8....With google crawl being less likely to crawl over 3 clicks deep, will pages not immediately linked get penalized ... It really comes down to user experience. If you lay out your site as if it was a gigantic story book where users will have to click "next page" all the time, then Google (or any other search engine) might not have ...


1

Interesting question! To me it really seems like a matter of crawl budget vs. link juice. Is it perhaps possible to block certain parts of the result pages set? For instance those with particular parameters that nobody links to? Does it seem like you're already having an issue with the crawl budget on your site? If not, then keeping the link juice might be ...


1

http://tools.seobook.com/robots-txt/ Google does not support the crawl delay command directly, but you can lower your crawl priority inside Google Webmaster Central.


1

As I understand you have many pages that are less important that shouldn't be crawled (that often). I don't know if it will help removing the pages from the sitemap. The sitemap is not a selection of pages that will be crawled. When a search engine finds links outside the sitemap, they will just as well be crawled. Did you consider assigning a lower ...


1

How often Google indexes a site for new content is based on an algorithmic evaluation undertaken by Google. The same process is followed for both blogs as well as standard websites. The first step is that Google will undertake an initial crawl of your site. This can be from Google finding a link on another site that links back to your site, or from you as a ...


1

May be Google check your robots.txt frequently, because you have moved your site content to new location. There is no any way to control robots.txt crawling rate. But, If your robots.txt did not blocked anything like any directory, then don't use robots.txt at all. For blocking other bots, you can use httacess. I have seen most of sites, don't use robots....


1

I can confirm that Google crawls more when your site is faster and less when you site is slower. In Google Webmaster Tools, the "Crawl rate" graphs will illustrate this perfectly. "Time spent downloading a page" will almost always be inversely proportional to the "Pages crawled per day" and "Kilobytes downloaded per day" graphs. Your Googlebot crawl ...


1

Googlebot crawls the web on a continuous basis. It also doesn't crawl your entire site at once. It may crawl some pages one day, some page the next, and some pages next week. Googlebot doesn't schedule crawls of site, rather it schedules crawls of individual pages. It may crawl your home page every day, but your deep pages every month. The rate at ...


Only top voted, non community-wiki answers of a minimum length are eligible