Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm starting a new website that will (hopefully) have a lot of user generated pictures. I'm trying to figure out the best way to store and serve these pictures.

The CMS I'm using (umbraco) has a media library that puts a folder on the server for each image. Inside of there you can have different sizes of that same image. That folder has an ID on it and the database has additional information for that image along with the ID of the folder.

This works great for small sites, but what if the pictures get up to 10,000, 100,000 or 1,000,000? It seems like the lookup on the directory would take a long time to find the correct folder. I'm on windows 2008 if that makes a difference.

I'm not so worried about load. I can load balance my server pretty easily and replicate the images across the servers. The nature of the site won't have a lot of users on it either, but it could have a lot of pics.

Thanks.

-Nate

EDIT
After some thought I think I'm going to create a directory for each user under a root image folder then have user's pictures under that. I would be pretty stoked if I had even 5,000 users, so that shouldn't be too bad of a linear lookup. If it does get slow I will break it down into folders like /media/a/adam/image123.png.

If it ever gets really big I will expand the above method to build a bigger tree. That would take a LOT of content though.

share|improve this question

migrated from programmers.stackexchange.com Mar 21 '11 at 11:04

This question came from our site for professional programmers interested in conceptual questions about software development.

    
Are you looking for SEO benefits, like the ability for people to be able to google your images? If so, then you'll want to name the images using keywords. Otherwise, use whatever is easiest. –  jmort253 Mar 20 '11 at 3:53
    
O_o 10,000 ,100,000 or 1,000,000 pictures? Why does only one thing come to my mind. –  AdityaGameProgrammer Mar 20 '11 at 7:23
    
It's actually a site to record workouts through some software. Each workout gets saved as a picture in a few different sizes. If you even get 5,000 people working out a few times a week it scales up pretty quickly. –  Nate Mar 20 '11 at 16:21

4 Answers 4

Such problems of scale have efficient solutions on cloud platforms such as Azure or AWS. Even though we are talking about a local filesystem, the same concepts can be applied here. Three things to consider in your solution:

1) Remove any association between any attributes of the resource and its physical location. For example, avoid using file names, titles, computed hash values etc. to determine where the photo will reside.

2) Use a sharding algorithm suitable for the desired scale and the available resources to determine the physical placement of photos. For example, if you have three volumes of equal size, then your sharding algorithm might be designed to distribute photos in sub-folders on the volumes such that the space usage on those volumes is balanced. You may also distribute photos in such a manner that reads can be more performant by using multiple disk spindles. It is best to keep things simple...numerical sequences of folder names work best. Here's something purely for illustrative purposes and not intended to be a recommendation:

00000000 / 000 to 999 / 000.jpg to 999.jpg

00000001 / 000 to 999 / 000.jpg to 999.jpg

3) Use database table to store the meta data and a pointer to the physical file(s).

Using this approach, you will be able to scale this to large numbers of images with good performance.

share|improve this answer
    
Thanks! Does Azure and AWS do that sharding for you? Or do you just get the multiple disks to store the files on and you still have to do the directory structure? –  Nate Mar 20 '11 at 16:18
1  
Since you're using .NET, Azure is the more logical choice so I'll focus on that. On Azure, you have Tables and Blobs. A table is a key/object store and a Blob is a binary store. You would not have to think about the sharding aspect as the details of the physical infrastructure are abstracted from you. You would add a photo to a blob and then add a row to a table with the unique identifier you gave to the blob. Here's a sample Azure photo app to help you get an overview: codeproject.com/Articles/106806/… –  Nik Kalyani Mar 20 '11 at 17:33

Having each image in its own directory really is overkill, and you're right, it'll start causing performance problems when you get tons of images in there. The actual point at which you reach this depends on the OS. But it can slow things down significantly.

Since you're tracking the image in a database, you can use the unique row id for the image name. So for the image at row 1, save the name as '1.jpg'. If you need to track different versions or revisions, you can name them like '1-resized.jpg', '1-original.jpg', etc. In the db, you can store the original file name and/or file extension being used.

If you expect a lot of images, break them down into multiple directories based on the id using some expression. For example, truncate( id / 1000), which would put the first 1000 images into directory 0 ('0/1.jpg', '0/1-resized.jpg'), the next 1000 into '1', etc. Thus When you need to reference image #15025, you know that image is '15/15025.jpg'. (if you want to be slick, pad the directory name with zeros so they're sortable)

If you end up with a million images, they'd be broken down into 1000 directories of 1000 images each, which is still navagatable via the command line if you need to manually manage things.

share|improve this answer

Generate a hash value for each picture, based on the contents of the picture (like SHA-1 or SHA-2) and separate the directory structure based on the beginning value of the hash (i.e. 64 directories covering a range of the hash values:

/images/00-03 /images/04-07 /images/08-0B ... (etc)

OR another breakdown

/images/0000 /images/0001 /images/0002 ... /images/000A ... (etc)

Some file would be named 0003ABC2EFA23.png. It would be found in directory: /images/0003

The number of the directory would represent the first digits of the hash value. You can configure it to use a wider OR smaller range of hash values. This allows you to break up the files into separate directories and quickly find the file you want based on this hash.

NOTE: make sure you consider collision resolution of the hash (because it can and probably will happen). Something like 0003ABC2EFA23-01.png for the first collision, 0003ABC2EFA23-02.png for the second.

share|improve this answer
    
Thanks for the answer. I was hoping I didn't have to do this but I think it might be the only thing I can do. Is this pretty standard for managing files for large websites? Seems like there should be a .NET library for this thing already...I'll look around. –  Nate Mar 20 '11 at 3:31
    
I do this today for a large number of attachments (not just image files). Others may have ideas too, but this is what we do to separate the file system into manageable parts. –  jmquigley Mar 20 '11 at 3:33

You mentioned that information was stored in the database... why not search the database then go to the folder directly?

share|improve this answer
    
So maybe I dont understand it correctly, but let's say we serve up something like this: website.com/images/12341231.png Is it a linear search through the folder for that file? That's what I'm afraid of. –  Nate Mar 20 '11 at 3:19
    
I think his issue is the strategy of how to break up the file system into manageable blocks and not the list of actual files. He doesn't want to have all the files in one directory because of the directory access overhead (which is a real concern, I have worked with directories with a huge number of files before and it cause some serious performance issues). –  jmquigley Mar 20 '11 at 3:23
    
If I understood correctly, they weren't all in the same folder... each image had their own folder with different sizes and other information. –  Kenneth Mar 20 '11 at 3:29
    
You still get that same big linear look up if it's all on one level though, right? –  Nate Mar 20 '11 at 3:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.