All Questions
Tagged with web-crawlers crawl-rate
25 questions
0
votes
0
answers
21
views
Exceed crawl limits for a website [duplicate]
In previous month i created a lot of pages (> 200K), and google indexed it. After some time a relized that i don't need some of these pages and created a new ones, but google don't indexing it. I ...
- 23
1
vote
0
answers
18
views
Website pages crawled entirely, thereafter only a 50% of pages are [duplicate]
Any technical or implementation tips on how to improve the crawling of site pages, when a given site used to have most pages crawled on its launch and thereafter only > 50% of them are. This ...
- 23
2
votes
1
answer
398
views
Can I set Cawl-delay in robots.txt to less than one second (like 0.1)?
I would like to start using the Crawl-delay on the robots.txt and saw that the value is in seconds. Is there a way to set it up below 1 second. Maybe 0.1?
My goal is to allow crawlers not to run over ...
- 123
2
votes
0
answers
30
views
Googlebot increased the number of requests for a certain URL even though it has no links
As part of regular Log File Analysis, I noticed in a URL that Google_bot was crawling like crazy. This URL is an Iframe that embed in a page. My thought was maybe this page gets a new link from ...
- 51
1
vote
1
answer
448
views
I have 2.1 million pages, how to get crawled by googlebot more frequently? [duplicate]
I have a project which aggregates public information on companies, startups, founders etc. for a total of >2 million tracked entities worldwide.
Yes, every single page is high quality, manually ...
- 111
1
vote
2
answers
2k
views
Too many hits from GoogleBot
The last week I have about 500.000 hits/day from googlebot in one of my websites. As a result, the server has serious issues to handle the rest of the requests.
What I have done
Verified that it is ...
- 193
2
votes
2
answers
53
views
Will lengthening our URLs change how Googlebot crawls our site?
Currently we are using URL fetched from main title of article (55 characters). We are planning to increase title of article up to 100 characters which tends to increase length of URL to 100+.
Will ...
- 184
3
votes
1
answer
441
views
How to change Google crawl rate in new Google search console?
Google has now completely removed the old search console (https://searchengineland.com/the-old-google-search-console-is-no-longer-available-321650) where you used to be able to request a custom crawl ...
- 163
3
votes
3
answers
299
views
Stop googlebot crawling URL more than once?
I have a site that usually creates a few thousand pages a day, which don't change after they have been created. Recently my dedicated server has crashed due to googlebot crawling the site too often. ...
- 33
-1
votes
1
answer
72
views
Maximum time to cache HTTPS URL after HTTP to HTTPS migration?
We have migrated our site from HTTP to HTTPS. How much time will the Google crawler take to cache all our HTTPS pages. We have more than 50K pages in our website.
- 77
2
votes
2
answers
311
views
Pagination and crawl depths
On a website with ~40 blogs, we have recently switched on pagination meanings blogs are on page 1-8.
With google crawl being less likely to crawl over 3 clicks deep, will pages not immediately linked ...
- 23
1
vote
1
answer
148
views
Block Google from crawling low value pages
I recently analysed the week-on-week crawl request made by google bot to my server. Here is the graph for the same.
Duplicate requests are when more than 1 request was made by google-bot for the same ...
- 446
2
votes
1
answer
437
views
What do I need to be aware of when running a web crawler?
I am getting into more holistic data analysis and would like to use a web crawler to extract data from websites for use in my analytics. To be clear, I don't want to mirror data and republish, at most ...
- 123
2
votes
2
answers
73
views
How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains?
After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each ...
- 13.6k
1
vote
2
answers
270
views
Can you find out when Google will next send Googlebot to crawl your website?
I can find the last date when Google crawl of my website but I can't find the next crawl date.
Is there some place I can look to find out when Googlebot is scheduled to return?
- 11
1
vote
2
answers
73
views
How to control old indexed page crawling rate
I have a site where user can create their profile page with their specific URL. Each specific URL will be available to public which means that it will also be ready for search engines too to crawl. ...
- 127
0
votes
1
answer
392
views
Understanding the maximum hit-rate supported by a web-server
I would like to crawl a publicly available site (and one that's legal to crawl) for a personal project. From a brief trial of the crawler, I gathered that my program hits the server with a new ...
- 101
1
vote
1
answer
815
views
Why does bingbot access my sites the most even if I set crawl rate to the minmum?
Recently my sites got overloaded and it turned out bingbot was the culprit. I set the crawl rate for the minimum for my sites in bing webmaster tools, but regardless bingbot has the most access (about ...
- 498
1
vote
1
answer
2k
views
Sudden decrease in pages crawled per day
During the last few days, Googlebot has crawled much less page in my site than it normally does. I'm not seeing any new errors, and haven't identified a reason for this. Is such a hugh and sudden ...
- 2,018
4
votes
2
answers
721
views
Controlling robot crawling concurrency using robots.txt and the undocumented setting "host-load"
I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
- 141
2
votes
3
answers
2k
views
How to decrease the frequency of Google/Search-engine bots crawling my site?
My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots.
My site generates over 1000 posts each day, ...
- 183
2
votes
1
answer
835
views
Page load speeds effect on crawl rate
We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However it'...
- 21
2
votes
1
answer
732
views
How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?
I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page ...
- 1,101
2
votes
2
answers
261
views
Will setting a custom crawl rate in Google Webmaster Tools affects index or have other negative consequences?
My site was running well for the past three years. Recently Googlebot's request my server got increased. and at one time I got network unreachable error. When I contacted my hosting company they asked ...
- 745
3
votes
2
answers
152
views
How to set crawl rate in Google Webmaster Tools for seven days or a month?
My web host is blocking IPs which make many requests. This includes Googlebot. I found this by checking my webmaster tools account. There it said "network unreachable" for all the URLs which are ...
- 745