3

I am currently trying to find a host for my website. The website allows people to host projects in a somewhat similar way to this. I would rather not reveal more about what it is until I release it. The problem is, that with users uploading files for their mods/projects/etc I would quickly reach any inode limit, especially with my current folder structure.

http://www.example.com/files/HOrrZCINYUoTxPVzaRrRFYUQRDxPPRun/wiD4Pj38Tkq/SomeFileUploadedWithTheProperNameStillHere.jar.

As you can see from that, it is a quite long URL. The first set of jumble is the project id (every project has one), it stays the same for all files of the same project. The second set of jumble is the file id, randomly generated for each file. There is a file id so that files of the same name can be uploaded to each project (Example would if someone uploads 2 MyProject.jar files, one for version 1 and another for version 2). So far I cannot find any hosts that allow for unlimited inodes, every host has a 25,000-500,000 limit (hence too small). Even if that was big enough, I would prefer not to have the imminent loom of inode limit bypassing over my head all the time (It would drive me insane). I would like either a link to a host with unlimited MySQL, disk space, and inodes. Or another way to do this. I was thinking about storing the files in a MySQL database, however it seems like most hosts limit those too (Including the database's files in the inode count).

2

The inode limit is only set on things such as reseller/shared hosting. If you have a VPS you can expect to be limited to around 1,000,000 inode and if you have a dedicated server you should be limited. However as you know, this will cost more depending on your needs.

They have to limit the inodes, if they didn't limit this you would be looking at "unlimited resources". Everytime you upload an item/make an insert into the database you will use an inode. Think about it, how big are your files going to be? Basing this just on uploads and not on database inserts if your limit is 25,000 and your average upload is 5mb this would equate to 125000mb which is about 122gb this is on the bottom scale. If you look at the size of the MySQL jar library which is 6.1mb and take a mid-range of the limit you have found, call it 240,000 inodes then thats 1464000mb, which is about 1429gb which is 1.39tb.

Surely if you are looking at something that would use up this amount of resources this quickly you would need to look at multiple servers, multiple locations, dedicated database servers and alot of bandwidth. The limit is all about not eating up server resources but if you do wish to implement this it maybe a case where the best possible solution is for you to host this yourself, but bare in mind for constant uploading of large files you will require a very good upload speed connection to the server.

  • I am planning on using shared hosting from some website such as hostgator, bluehost, ipage, etc. I am looking for "unlimited resources" :P. Also: I am planning on using cloudflare to cache the large files, so bandwidth is really not so much of an issue, but I still demand unlimited (I would go crazy worrying without it). Hosting it myself (and buying slots in a datacenter, correct me if this is not what you mean) is simply too expensive. – john01dav Apr 15 '14 at 18:33
  • Caching could help to reduce the use of inodes, however database side if you are inserting there's not much you can do about it. However if you are selecting which I doubt this would use an inode you could implement memcached or something similar, to be brutley honest you may not have a choice in the matter of using dedicated or your own server. – Liam Sorsby Apr 16 '14 at 6:26
  • I actually contacted several hosts, they are saying that if I use InnoDB then an entire table is stored in a single file, so (in theroy, correct me if I am wrong) I should be able to just store everything in a blob and retrieve it with php. How would (cloudflare) caching reduce the number of inodes (look it up if you do not know what it is)? Memcached is not supported my my host (I think), but it should not be a problem because cloudflare serves the files and only does a real request once every 2-4 hours. – john01dav Apr 16 '14 at 6:49
  • I've seen cloud flare which is a cdn for static resources however as soon as you start making uploads file writes this will then be handles by your server. Cloud flare will serve static content such as HTML, js, CSS and static images but nothing else. With regards to using blob bare in mind that you will increase server load as more content is written to the blob especially over mobile as if it increase to say 10mb it will take an age for the device to getthe required information – Liam Sorsby Apr 16 '14 at 7:26
  • You can use page rules in cloudflare to force it to cache anything. What business do mobiles have download minecraft (pc) mods? It should not need to be downloading the blob with just browsing the site, as those would be omitted from the mysql queries. – john01dav Apr 16 '14 at 7:31
2

Outside of "cloud" hosting, no resources are unlimited, regardless of what a marketing department says. If you need a large amount of resources, expect to pay for a large amount of resources. Shared hosting is economy hosting, and intended to be a cheap service for users who are just getting started with hosting, and want to try out a simple site or two. Don't expect to produce a large application on this type of hosting.

There are many reasons why a host will limit the amount of inodes available. One reason is that they are actually not unlimited on a physical server. Many of the popular filesystems, such as ext3 and ext4, have inode limits. With those in particular, you need to determine the total number of inodes, and thus the static space required, when you crate the filesytem. Each inode takes up space. Therefore, the more inodes you use, the less disk space will be availalbe. Fortunately, these are small, so you can allocate many of them, and that's fine as long as they are used reasonably. These are typically a number determined as a fraction of the total disk space available. On shared environments, these must be distributed among the users on the server so that nobody can crash the whole server by creating tons of empty files.

Another reason that a host may limit inodes is because they don't want to encourage certain types of storage. Email is an example where you typically can have very many small files, and large amounts of inode usage can indicate a spammer. Also, consider that some hosts will also not like strange data storage schemes, and could be a sign that something problematic is happening. Or perhaps they want to create some reasonable way of limiting "unlimited" disk space usage.

Usually, unless you have a lot of tiny files, you couldn't use that many inodes without also using a lot of space, and also a lot of other resources as well. If your files are small, then consider some other data storage method, such as a database. If the data is actually larger, then expect to pay for more disk usage.

Finally, if your main concern isn't the cost, but wanting to be sure that your application is scalable, then you could look into some of the more scalable solutions. For instance, there's the Amazon elastic hosting and also the S3 storage buckets. They are designed to allow for increasing resources as needed. However, you will need still need to pay for all of your usage, so if you do end up using a lot of resources, it will still cost money.

0

I can't find any reliable host with no limit on inodes on there VPS plans. I was having this inodes problem on Hostgator and in-fact they have very low inodes limit if you are a reseller and hosting multiple sites. So after researching a lot I decided to go with Known Host which also has inodes limit but that is really very high and with different VPS packages it also changes. After using hostgator I was very reluctant to move to any other host but I found Known Host really very good.

  • They have both bandwidth, and disk space limitations. – john01dav Apr 17 '14 at 7:50
0

I have had to shut down 2 websites that were doing very well as far as members and traffic. I was starting to look into cashing in on advertisement until I got an email warning me that I reached my max for storage and inodes. I lost all my members and couldn't start over.

I realized that the only way to have a website with unlimited inodes, storage and resources similar to facebook, youtube and other social media is to have my own servers set up at a warehouse with engineers on payroll to support software upgrades and server issues.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.