The web-crawlers tag has no wiki summary.
4
votes
2answers
200 views
Is it possible for web crawlers to see static pages without following a link to them?
If I create a static page on a domain (http://www.domain.com/page.html), can a crawler still see it if there aren't any links to it anywhere on the site?
3
votes
2answers
247 views
How often are sitemap.xml checked for updates by crawlers?
How often do crawlers check for updates in submitted sitemap.xml? Every hour, day, week, month, more?
9
votes
1answer
506 views
How did Google manage to crawl my 403 pages?
I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
