Questions tagged [crawl-rate]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
49 views

I have 2.1 million pages, how to get crawled by googlebot more frequently? [duplicate]

I have a project which aggregates public information on companies, startups, founders etc. for a total of >2 million tracked entities worldwide. Yes, every single page is high quality, manually ...
1
vote
1answer
112 views

Too many hits from GoogleBot

The last week I have about 500.000 hits/day from googlebot in one of my websites. As a result, the server has serious issues to handle the rest of the requests. What I have done Verified that it is ...
3
votes
2answers
101 views

How to limit Google crawling certain URL patterns?

recently we have updated our e-commerce website, where each product category has product filter. There is around 30 attributes which can be used for filtering products shown in category. These ...
0
votes
1answer
22 views

Google Crawl rate & the log file mystery

According to Google Search Console for a website I am working on, Googlebot crawls ~5000 pages per day (min 2500, max 8500). However, when looking at the Apache log files, GoogleBot only shows up ~10 ...
2
votes
2answers
39 views

Will lengthening our URLs change how Googlebot crawls our site?

Currently we are using URL fetched from main title of article (55 characters). We are planning to increase title of article up to 100 characters which tends to increase length of URL to 100+. Will ...
7
votes
2answers
4k views

Googlebot crawls my page too often

I have one article in my blog that crawled by Google too often. My blog is new and I have about 100 articles in my blog with 50-80 page views (including Google bot). But I have 1 article that have ...
3
votes
1answer
234 views

How to change Google crawl rate in new Google search console?

Google has now completely removed the old search console (https://searchengineland.com/the-old-google-search-console-is-no-longer-available-321650) where you used to be able to request a custom crawl ...
2
votes
3answers
91 views

Stop googlebot crawling URL more than once?

I have a site that usually creates a few thousand pages a day, which don't change after they have been created. Recently my dedicated server has crashed due to googlebot crawling the site too often. ...
1
vote
1answer
226 views

Is Applebot following Crawl-Delay?

Applebot have been crawling my website like crazy for weeks now, and it does not seems to slow down. Does it follow Crawl-Delay from robots.txt, or not?
1
vote
2answers
140 views

Yandex ignores Crawl-delay?

I try to get the Yandex Bot under control and I'm close to block it. After 2 months of robots.txt adjustments, it still keeps hammering my server from multiple IPs. User-Agent: * Crawl-delay: 4 ...
1
vote
1answer
49 views

How to get Google to crawl our product pages as soon as they are updated for in-stock or out-of-stock?

We have an automobile website where we have different cars and bike models. We have product pages for every car/bike. Each page has the products price, Offers, reviews , news, images and videos. ...
-1
votes
1answer
71 views

Maximum time to cache HTTPS URL after HTTP to HTTPS migration?

We have migrated our site from HTTP to HTTPS. How much time will the Google crawler take to cache all our HTTPS pages. We have more than 50K pages in our website.
2
votes
2answers
217 views

Pagination and crawl depths

On a website with ~40 blogs, we have recently switched on pagination meanings blogs are on page 1-8. With google crawl being less likely to crawl over 3 clicks deep, will pages not immediately linked ...
1
vote
1answer
493 views

Internal Search Results: NOINDEX or robots.txt Blocking?

We have our own internal search results pages for our download section. These results are currently crawlable but with NOINDEX meta tags so they don't appear in the search index. People link to ...
1
vote
1answer
2k views

“Crawl-delay” rule ignored by googlebot

In Google Webmaster Tools this warning showing for site: The yellow signal indicating "rule ignored by googlebot". How can I fix those warning?
1
vote
1answer
117 views

Block Google from crawling low value pages

I recently analysed the week-on-week crawl request made by google bot to my server. Here is the graph for the same. Duplicate requests are when more than 1 request was made by google-bot for the same ...
2
votes
1answer
499 views

Significant decrease in crawl rate

Below is the graph for my website which has seen a significant decrease in the crawl rate(pages crawled per day) Here is the graph for the time to download the page I wanted to know the reason for ...
1
vote
2answers
771 views

Why Does It Take So Long For Google to Crawl or Re-Crawl My Site

My site hasn't yet been crawled and it has been up for (XX) days or hasn't been re-crawled for (XX) days. Why it is taking so long and how can I speed it up. (Created for the large number of ...
2
votes
1answer
68 views

Googlebot takes a week to index my blog, how do I increase the rate?

Right now I see it comes after 7 days. I am using a Wordpress plugin for SiteMap generator and just changed it's frequency from monthly to weekly. I know there's a manual submission option available ...
0
votes
1answer
20 views

How to GEt Crawled Frequently [duplicate]

I wanted to know how to get crawled your Web Page Frequently to Different Search Enginges. I want my Web to Get Crawled to Google Every Frequenct Day? Can Anyone help me out.
2
votes
1answer
140 views

What do I need to be aware of when running a web crawler?

I am getting into more holistic data analysis and would like to use a web crawler to extract data from websites for use in my analytics. To be clear, I don't want to mirror data and republish, at most ...
1
vote
1answer
181 views

Why is this mobile/image Googlebot crawl rate so fast? (10-30 req per sec)

This morning in logs I noticed a user got rate limited by our throttler 3 times within 1 second (429 too many requests error) at 3:30:11 AM EST: Mozilla/5.0 (Linux; Android 6.0; GCE x86 phone Build/...
2
votes
2answers
61 views

How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains?

After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each ...
1
vote
2answers
237 views

Can you find out when Google will next send Googlebot to crawl your website?

I can find the last date when Google crawl of my website but I can't find the next crawl date. Is there some place I can look to find out when Googlebot is scheduled to return?
0
votes
0answers
155 views

Indexed URL is very low comparing submitted URL in google

I have 84 URL in my website. Till 2 weeks back all the URL is indexed in Google. I have verified it in Google webmaster tools. But it was slightly decreased day by day. Now Google webmaster tools ...
1
vote
2answers
62 views

How to control old indexed page crawling rate

I have a site where user can create their profile page with their specific URL. Each specific URL will be available to public which means that it will also be ready for search engines too to crawl. ...
2
votes
1answer
104 views

Bing Crawl Control Time Zone

The Crawl Control in Bing Webmaster Tools allows you to "tell Bingbot to crawl your site faster or slower than the normal crawl rate for each of the 24 hours in the day" according to http://www.bing....
0
votes
1answer
267 views

Understanding the maximum hit-rate supported by a web-server

I would like to crawl a publicly available site (and one that's legal to crawl) for a personal project. From a brief trial of the crawler, I gathered that my program hits the server with a new ...
0
votes
4answers
2k views

Pages Crawled Per Day in Google Webmaster Tools

What determines the pages crawled per day graph in Google webmaster tools? It shows that It has a maximum of 2701 pages. However, the maximum number of pages on my website is not more than 300. How ...
1
vote
2answers
108 views

Should you let Googlebot crawl and index your entire site before updating it again?

I have heard from various people that "Google has scanned x% of the site". Some have told me at times not to upload anything, nor do any changes on the site "since Google hasn't finished scanning the ...
1
vote
1answer
459 views

Why does bingbot access my sites the most even if I set crawl rate to the minmum?

Recently my sites got overloaded and it turned out bingbot was the culprit. I set the crawl rate for the minimum for my sites in bing webmaster tools, but regardless bingbot has the most access (about ...
1
vote
3answers
325 views

How to make Google index all sitemap.xml records? [duplicate]

I have a sitemap.xml index fine with 3 XML files in it with around 120 pages for indexing. A year passed and I still have 1/3 of it indexed. All the most important pages for indexing are in 1st ...
1
vote
1answer
2k views

Sudden decrease in pages crawled per day

During the last few days, Googlebot has crawled much less page in my site than it normally does. I'm not seeing any new errors, and haven't identified a reason for this. Is such a hugh and sudden ...
4
votes
2answers
591 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
2
votes
3answers
2k views

How to decrease the frequency of Google/Search-engine bots crawling my site?

My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots. My site generates over 1000 posts each day, ...
2
votes
1answer
805 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However it'...
0
votes
2answers
146 views

Does description change impact the google ranking and/or crawl rates?

I am thinking about adding some randomization to my shopping cart descriptions and pricing. I am not talking about meta-description, but about actual product descriptions. Would content change ...
2
votes
1answer
697 views

How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?

I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page ...
3
votes
2answers
216 views

Google crawl rate is climbing but still no indexing?

This might belong on another stack site but idk. Soooo.... A year ago I started two sites that I didn't want indexed by Google or any other search engine since both sites were being developed based on ...
2
votes
2answers
254 views

Will setting a custom crawl rate in Google Webmaster Tools affects index or have other negative consequences?

My site was running well for the past three years. Recently Googlebot's request my server got increased. and at one time I got network unreachable error. When I contacted my hosting company they asked ...
3
votes
2answers
132 views

How to set crawl rate in Google Webmaster Tools for seven days or a month?

My web host is blocking IPs which make many requests. This includes Googlebot. I found this by checking my webmaster tools account. There it said "network unreachable" for all the URLs which are ...