What is a recommended maximum number of files in a directory on your webserver?

Question

I have directory that contains 400,000+ files. Potentially the directory could easily contain ~30,000,000 files.

1. Is that a good idea or should I rather chop it up in smaller directories like this:

   /images/1/
   /images/2/
   /images/3/
   /images/4/
   etc.
   

2. What size should I make the smaller directories? Would 100,000 files in each directory be a good idea?

Answer 1

I am going to give this a shot.

From a filesystem point of view:

FAT32:

• Maximum number of files: 268,173,300
• Maximum number of files per directory: 2¹⁶ - 1 (65,535)
• Maximum file size: 2 GiB - 1 without LFS, 4 GiB - 1 with

NTFS:

• Maximum number of files: 2³² - 1 (4,294,967,295)
• Maximum file size
  • Implementation: 2⁴⁴ - 2⁶ bytes (16 TiB - 64 KiB)
  • Theoretical: 2⁶⁴ - 2⁶ bytes (16 EiB - 64 KiB)
• Maximum volume size
  • Implementation: 2³² - 1 clusters (256 TiB - 64 KiB)
  • Theoretical: 2⁶⁴ - 1 clusters

ext2:

• Maximum number of files: 10¹⁸
• Maximum number of files per directory: ~1.3 × 10²⁰ (performance issues past 10,000)
• Maximum file size
  • 16 GiB (block size of 1 KiB)
  • 256 GiB (block size of 2 KiB)
  • 2 TiB (block size of 4 KiB)
  • 2 TiB (block size of 8 KiB)
• Maximum volume size
  • 4 TiB (block size of 1 KiB)
  • 8 TiB (block size of 2 KiB)
  • 16 TiB (block size of 4 KiB)
  • 32 TiB (block size of 8 KiB)

ext3:

• Maximum number of files: min(volumeSize / 2¹³, numberOfBlocks)
• Maximum file size: same as ext2
• Maximum volume size: same as ext2

ext4:

• Maximum number of files: 2³² - 1 (4,294,967,295)
• Maximum number of files per directory: unlimited
• Maximum file size: 2⁴⁴ - 1 bytes (16 TiB - 1)
• Maximum volume size: 2⁴⁸ - 1 bytes (256 TiB - 1)

Reference:
https://stackoverflow.com/questions/466521/how-many-files-can-i-put-in-a-directory

From a functionality point of view:

Keep in mind that on Linux if you have a directory with too many files, the shell may not be able to expand wildcards. I have this issue with a photo album hosted on Linux. It stores all the resized images in a single directory. While the file system can handle many files, the shell can't.

From a server speed point of view:

Too many files in one directory can cause load times to increase by seconds. Having too many directories can also increase load times. (Your server specs play a role in this)

From an SEO point of view.

I can understand not having proper image names for security reasons (assuming a user uploads photos and you have a rewrite in place) But you really should give a content relevant (sub)directory names for better search ranking and the ability to partially remove URL's to get to another spot on your server. (ie photos/outdoor/landscape/mountains)

In the end there is no one fits all but you can make a better informed decision based on the aforementioned.

Answer 2

I've tried to have directories with tens of thousands of files on ext3 and ext4 Linux. It eventually gets to the point where listing those files is prohibitively expensive. It can take a few seconds for ls to complete.

I would try do design for a max of 10,000 files per directory.

One way of doing so is to use the first couple characters of the file name as a directory. The git source control system takes this approach. See Why does git store objects in directories with the first two characters of the hash?. Git limits the number of hash revisions per directory to 6700.