Search engines only index URLs that have unique content on them. If your folder URL doesn't return any useful content, search engines won't index it. That will be the case if the folder URL:
- Returns a blank page
- Has a error code such as 403 Forbidden or 404 Not Found
- Redirects to some other URL
Search engines do try to discover content by crawling folders. If you use the URL /category/page
on your site, search engines may very well try to crawl /category/
to see if they can find content there.
If you are powering your website with static files, directory URLs may have content that is automatically generated by your web server. Most web servers are configured to serve a directory listing. That is often enough "content" for search engines to index. Further more, search engines will follow the links in that directory listing and find all the other files you have in the directory.
If you don't want your directory listing indexed you can add an index.html
to that folder to replace the directory listing with content of your own choosing. Alternately you can use Options -Indexes
on Apache to turn off the generation of directory indexes and instead serve an error page.
Search engines never index the same URL multiple times, even if they find multiple ways to get to the same URL. If you link to /category/
and search engines also try find it because they expect it from /category/page
they will still crawl and index that URL only once.
For directories there are some issues with URL canonicalization. All three of these URLs may return the same content:
/category
/category/
/category/index.html
Web servers usually redirect to add the slash to directory URLs. Even if you power the directory content with an index.html
file you shouldn't use index.html
in your links. Just link to the directory URL ending in the trailing slash.