I'm running a crawler on my website to test for broken links and such.
It starts by using a URL like www.domain.com
One curious thing is that it is showing directories with no internal links. For example, directory /example_dir/ is showing up in the crawl tree, but I can't find any internal link to that directory within the pages.
How could this be happening and is there a way to prevent it?