Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm running a crawler on my website to test for broken links and such.

It starts by using a URL like www.domain.com

One curious thing is that it is showing directories with no internal links. For example, directory /example_dir/ is showing up in the crawl tree, but I can't find any internal link to that directory within the pages.

How could this be happening and is there a way to prevent it?

share|improve this question
add comment

2 Answers

What tool are you using to crawl your site?

Crawlers typically find new pages by following links so the odds are you have a link pointing to those directories. It may not be intentional, such as a a dynamic link that is pulling up bad data but not throwing out an error. If you aren't using Xenu's Link Sleuth I recommend using it as it will tell you what pages had links that lead it to crawl those directories.

share|improve this answer
1  
It was Netsparker which is a web vulnerability scanner. Doesn't do broken links like I thought. I grabbed Xenu and that works great, but didn't show the directory in question. My guess is Netsparker is picking it up from the robots.txt file and that's the difference here. –  edeneye Nov 3 '11 at 19:03
add comment

My guess is that Jon is right, you must have a link somewhere. It might not show on the page but the spider is finding it.

Don't forget that code like this can happpen <a href="/my_dir/"></a>. Although it's blank to the user, it will be followed by the spider.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.