If I create a static page on a domain (http://www.domain.com/page.html), can a crawler still see it if there aren't any links to it anywhere on the site?
How often do crawlers check for updates in submitted sitemap.xml? Every hour, day, week, month, more?
I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...