Hot answers tagged web-crawlers
Google, conceptually, uses an HTML DOM parser. What this does is break any web page HTML down into its basic structure and each HTML tag is given an ID. This ID represents the order of the HTML tags from beginning to end, any dependency between HTML elements such as a li tag is dependent upon a ul tag, any parent-child relationship between HTML elements ...
The default directive for spiders is allow and when Google visits your robots.txt and sees the disallow command, it immediately knows that it isn't allowed to crawl the website anymore. The subsequent allows don't matter. Do note that when there are conflicting codes within the robots file or between the robots.txt and the Meta Robots tag of a page, spiders ...
If you have a new page, you probably have a link to it. Google visits your page often and will catch the link. If it's a matter of new content, that may take longer because I don't think Google examines content as often as it examines links. This is why they recommend a sitemap. When you have a significant change to a page, you can update your sitemap and, ...
From your comments on another answer, MJ12Bot is visiting your site less than once an hour (421 times in 25 days.) The best thing to do is to not worry about it. Crawl-Delay is useless for you because no crawler will obey a craw-delay so large.
Only top voted, non community-wiki answers of a minimum length are eligible