5

I have directory that contains 400,000+ files. Potentially the directory could easily contain ~30,000,000 files.

  1. Is that a good idea or should I rather chop it up in smaller directories like this:

    /images/1/
    /images/2/
    /images/3/
    /images/4/
    etc.
    
  2. What size should I make the smaller directories? Would 100,000 files in each directory be a good idea?

  • 1
    What is your main concern..load times, directory limits, SEO, directory structure? – norcal johnny Sep 23 '16 at 22:24
7

I am going to give this a shot.

From a filesystem point of view:

FAT32:

  • Maximum number of files: 268,173,300
  • Maximum number of files per directory: 216 - 1 (65,535)
  • Maximum file size: 2 GiB - 1 without LFS, 4 GiB - 1 with

NTFS:

  • Maximum number of files: 232 - 1 (4,294,967,295)
  • Maximum file size
    • Implementation: 244 - 26 bytes (16 TiB - 64 KiB)
    • Theoretical: 264 - 26 bytes (16 EiB - 64 KiB)
  • Maximum volume size
    • Implementation: 232 - 1 clusters (256 TiB - 64 KiB)
    • Theoretical: 264 - 1 clusters

ext2:

  • Maximum number of files: 1018
  • Maximum number of files per directory: ~1.3 × 1020 (performance issues past 10,000)
  • Maximum file size
    • 16 GiB (block size of 1 KiB)
    • 256 GiB (block size of 2 KiB)
    • 2 TiB (block size of 4 KiB)
    • 2 TiB (block size of 8 KiB)
  • Maximum volume size
    • 4 TiB (block size of 1 KiB)
    • 8 TiB (block size of 2 KiB)
    • 16 TiB (block size of 4 KiB)
    • 32 TiB (block size of 8 KiB)

ext3:

  • Maximum number of files: min(volumeSize / 213, numberOfBlocks)
  • Maximum file size: same as ext2
  • Maximum volume size: same as ext2

ext4:

  • Maximum number of files: 232 - 1 (4,294,967,295)
  • Maximum number of files per directory: unlimited
  • Maximum file size: 244 - 1 bytes (16 TiB - 1)
  • Maximum volume size: 248 - 1 bytes (256 TiB - 1)

Reference:
https://stackoverflow.com/questions/466521/how-many-files-can-i-put-in-a-directory

From a functionality point of view:

Keep in mind that on Linux if you have a directory with too many files, the shell may not be able to expand wildcards. I have this issue with a photo album hosted on Linux. It stores all the resized images in a single directory. While the file system can handle many files, the shell can't.

From a server speed point of view:

Too many files in one directory can cause load times to increase by seconds. Having too many directories can also increase load times. (Your server specs play a role in this)

From an SEO point of view.

I can understand not having proper image names for security reasons (assuming a user uploads photos and you have a rewrite in place) But you really should give a content relevant (sub)directory names for better search ranking and the ability to partially remove URL's to get to another spot on your server. (ie photos/outdoor/landscape/mountains)

In the end there is no one fits all but you can make a better informed decision based on the aforementioned.

  • 1
    It has been a while, and for the record, I do not know where the limit is, however, too many files in a directory will make it nearly impossibly slow for the file system to address any file in the file index. I experienced this years ago. Even if you do not hit that limit, other tools can consistently fail such as FTP. I have one directory that is huge but not that huge and FTP of any flavor or stripe will have a conniption and will fail every time. It is better not to push the limits. Cheers!! – closetnoc Sep 23 '16 at 23:31
  • Agreed, not pushing the limits. I personally think a couple of thousand is too many. Even on my mac if I open a folder with 2,000 photos, it starts to struggle. – norcal johnny Sep 23 '16 at 23:54
  • 1
    For the record- I like your answers! Cheers!! – closetnoc Sep 24 '16 at 0:07
  • I get worried about commenting off-topic but I do want to say I greatly appreciate your kind words! Cheers!! :) – norcal johnny Sep 24 '16 at 0:35
  • The max number of files in a directory on FAT32 ("65,535") is assuming you are using short 8.3 filenames. If you are using long filenames (LFNs) then this number can be greatly reduced, since each 13 byte filename chunk is stored in a separate directory entry – DocRoot Sep 27 '16 at 14:35
0

I've tried to have directories with tens of thousands of files on ext3 and ext4 Linux. It eventually gets to the point where listing those files is prohibitively expensive. It can take a few seconds for ls to complete.

I would try do design for a max of 10,000 files per directory.

One way of doing so is to use the first couple characters of the file name as a directory. The git source control system takes this approach. See Why does git store objects in directories with the first two characters of the hash?. Git limits the number of hash revisions per directory to 6700.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.