3

I have over 30,000 html files stored in www.example.com/docs, and these files are linked to each other. When it comes to displaying the content of those files on my site, I use the following code: HTML Include using Java

I learned today that my host allows a maximum of 10,000 files per folder. This means that I have start splitting up my files into different folders. I will probably create a number of folders like

  • www.example.com/docs1
  • www.example.com/docs2
  • www.example.com/docs3

This causes an issue because I have to amend the hyperlinks within all 30,000 files, and I have to update the URLs within pages that display documents within those folder.

I don't mind creating the folders and moving the files, but I am really dreading updating the links and URLs. Is there a solution I can leverage? I am thinking something .htaaccess related, but I am not knowledgeable enough.

EDIT: my files are in the following format doc[NUMBER].htm, so it look like this:

  • doc100.htm
  • doc101.htm
  • doc102.htm
  • What is the range/distribution of the number? It doesn't look zero padded. – MrWhite Feb 20 at 16:20
3

How you do this would depend on how these files are named. You could, for instance, divide these HTML files into subdirectories according to their first letter (assuming a fairly even/normal distribution). For example:

/docs-a
    aardvark.html
    apple.html
/docs-b
    banana.html
    bee.html
:
/docs-z
    zebra.html

Then, given a request for /docs/aardvark.html, it needs to be internally rewritten to /docs-a/aardvark.html (since that is the true location). This is relatively trivial to do using mod_rewrite in .htaccess. For example:

RewriteEngine On

RewriteRule ^docs/(([a-z]).*\.html)$ /docs-$2/$1 [L]

This assumes that all files start with a lowercase letter a-z and no one letter is excessively dominant (ie. well below the 10,000 limit).

Note that this is case-sensitive, a is different to A.

EDIT: my files are in the following format doc[NUMBER].htm

In this case, @Stephen's suggestion in comments sounds good - base the directory name on the last digit (to ensure an even distribution) and use 10 subdirectories: /docs-0 .. /docs-9.

RewriteRule ^docs/(doc.*([0-9])\.htm)$ /docs-$2/$1 [L]

UPDATE: Can it modified to be /docs/docs-0

Yes, however, you'll also need to change the RewriteRule pattern to be more restriction, to avoid a rewrite loop. For example:

RewriteRule ^docs/(doc\d*(\d)\.htm)$ /docs/docs-$2/$1 [L]

\d is a shorthand character class for digits. The same as [0-9].

  • Thank you very much for your solution. I have edited my original post to include the format of my documents. Is there anything can be done based on my format? – jcmaad Feb 20 at 16:12
  • 1
    Base it on the last digit of the number instead: RewriteRule ^docs/(.*([0-9])\.html)$ /docs-$2/$1 [L]. You will need docs-0 through docs-9 directories. I'd recommend the last digit rather than the first because for numbers the last digit is more evenly distributed. The beginning digits have a preference for low digits, one being the most common. – Stephen Ostermiller Feb 20 at 16:21
  • Awesome. I will try this out and report back. – jcmaad Feb 20 at 16:32
  • @StephenOstermiller That's a good idea! (I've updated my answer.) Although that does perhaps make the initial moving of files into the respective directories a little harder - perhaps write a script for that? – MrWhite Feb 20 at 16:35
  • 1
    mv docs/*0.htm docs/docs-0/ and nine other similar statements on the command line. Less work than the 26 for each letter. If you want a one liner to do them all: for i in 0 1 2 3 4 5 6 7 8 9; do mkdir -p docs/docs-$i; mv docs/*$i.htm docs/docs-$i/; done – Stephen Ostermiller Feb 20 at 20:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.