The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
1answer
25 views

Understanding the maximum hit-rate supported by a web-server

I would like to crawl a publicly available site (and one that's legal to crawl) for a personal project. From a brief trial of the crawler, I gathered that my program hits the server with a new ...
0
votes
2answers
44 views

Pages Crawled Per Day in Google Webmaster Tools

What determines the pages crawled per day graph in Google webmaster tools? It shows that It has a maximum of 2701 pages. However, the maximum number of pages on my website is not more than 300. How ...
1
vote
2answers
48 views

Should you let Googlebot crawl and index your entire site before updating it again?

I have heard from various people that "Google has scanned x% of the site". Some have told me at times not to upload anything, nor do any changes on the site "since Google hasn't finished scanning the ...
1
vote
1answer
29 views

Why does bingbot access my sites the most even if I set crawl rate to the minmum?

Recently my sites got overloaded and it turned out bingbot was the culprit. I set the crawl rate for the minimum for my sites in bing webmaster tools, but regardless bingbot has the most access (about ...
0
votes
3answers
104 views

How to make Google index all sitemap.xml records? [duplicate]

I have a sitemap.xml index fine with 3 XML files in it with around 120 pages for indexing. A year passed and I still have 1/3 of it indexed. All the most important pages for indexing are in 1st ...
1
vote
1answer
269 views

Sudden decrease in pages crawled per day

During the last few days, Googlebot has crawled much less page in my site than it normally does. I'm not seeing any new errors, and haven't identified a reason for this. Is such a hugh and sudden ...
3
votes
2answers
151 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
2
votes
3answers
136 views

How to decrease the frequency of Google/Search-engine bots crawling my site?

My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots. My site generates over 1000 posts each day, ...