I'm working on a research project and I have a question.
Say I would like to crawl all pages of a given site. In the case that my bot is blocked from accessing a certain portion of the site, I would need to know for sure that it has been blocked, and that there exists at least one portion of the site that has not been crawled. Is this technically feasible under the current protocol? In other words, I don't want my bot to be blocked in a deceptive manner, which would lead me to believe that the entire site has been crawled when in fact it hasn't.