Hot answers tagged web-crawlers
There is no standard. You will have to consider how robust the site is. Multiple servers and lots of bandwidth would be less taxed than a single server with less bandwidth. Also consider that some are paying for the bandwidth they use rather than having a set amount rated on a per second basis (like we are used to with DSL and cable). This would mean that ...
Simple answer, yes, Bing and DuckDuckGo support this. This article suggests bing does support it. The site imelda-immobilien.de has no "static" pages, they're all ajax driven. A search on Bing and DuckDuckGo shows multiple ajax urls in the results. Yahoo is powered by bing, and so their results show the same.
Only top voted, non community-wiki answers of a minimum length are eligible