Tagged Questions

The tag has no wiki summary.

learn more… | top users | synonyms

1
vote
1answer
6 views

Bing Crawl Control Time Zone

The Crawl Control in Bing Webmaster Tools allows you to "tell Bingbot to crawl your site faster or slower than the normal crawl rate for each of the 24 hours in the day" according to ...
0
votes
1answer
29 views

Understanding the maximum hit-rate supported by a web-server

I would like to crawl a publicly available site (and one that's legal to crawl) for a personal project. From a brief trial of the crawler, I gathered that my program hits the server with a new ...
0
votes
2answers
162 views

Pages Crawled Per Day in Google Webmaster Tools

What determines the pages crawled per day graph in Google webmaster tools? It shows that It has a maximum of 2701 pages. However, the maximum number of pages on my website is not more than 300. How ...
1
vote
2answers
56 views

Should you let Googlebot crawl and index your entire site before updating it again?

I have heard from various people that "Google has scanned x% of the site". Some have told me at times not to upload anything, nor do any changes on the site "since Google hasn't finished scanning the ...
1
vote
1answer
46 views

Why does bingbot access my sites the most even if I set crawl rate to the minmum?

Recently my sites got overloaded and it turned out bingbot was the culprit. I set the crawl rate for the minimum for my sites in bing webmaster tools, but regardless bingbot has the most access (about ...
0
votes
3answers
116 views

How to make Google index all sitemap.xml records? [duplicate]

I have a sitemap.xml index fine with 3 XML files in it with around 120 pages for indexing. A year passed and I still have 1/3 of it indexed. All the most important pages for indexing are in 1st ...
1
vote
1answer
380 views

Sudden decrease in pages crawled per day

During the last few days, Googlebot has crawled much less page in my site than it normally does. I'm not seeing any new errors, and haven't identified a reason for this. Is such a hugh and sudden ...
3
votes
2answers
172 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
2
votes
3answers
174 views

How to decrease the frequency of Google/Search-engine bots crawling my site?

My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots. My site generates over 1000 posts each day, ...