Creating a board with thousands of forums

Question

I would like to create a message board that has separate forums for each movie, each actor, each director, etc. With modern forum software, is having many thousands of forums like this impractical for technical reasons?

1) I've heard that in, say, phpBB, the more forums you have, the greater the load on your database server. Why should that be? Why would there be more load on the server if you have 1000 boards each with 1 post, as opposed to 1 board with 1000 posts?

2) Also regarding phpBB, I've heard that you'll run into problems if you get over 1000 forums, because the permission checks back to the database will start to really slow things down. Again, why should this be the case?

For a purely database-specific perspective, if I have a table for each forum, I don't see what the problem is with having many thousands of forums. Are the problems above specific to phpBB? If so, is there forum software which would not have a problem with having thousands of boards?

(By the way, I realize there are logistic problems with having thousands of forums, which is why I would also want to make a front page which allows users to easily see the most recent posts in a few recently-visited forums. I'm not sure if the existing forum software will allow me to do this. Ideally I would prefer to write my own forum software, but that's a crazy amount of work for a small non-commercial project.)

Answer 1

With modern forum software, is having many thousands of forums like this (using one copy of the forum software package for all forums) impractical for technical reasons?

Part of the answer is dependent on the forum software. If the software is poorly written, it is possible that it makes alot of disk accesses and alot of database requests, etc.

More importantly, the server logs are where you will look for answers. If you try your experiment, check your database access logs, and web server access logs to see what is being requested the most/least and at what times. Multiple requests in the same second is a clue that someone may be trying to hack the server.

Just think of running 1000 forums like you running a dedicated server with 1000 websites on it. The effect is the same in both cases. Slowness is really noticed when the number of requests made to the server exceeds the connections the server has available, and its noticed more if the server is run on a very old computer.

Answer 2

I think I see where you are going with this. You talk about having a table for each forum, this is not all that common at all in forum engines. To give you a very broad strokes run down on the database schema for the "average" forum software you have 1 table containing a list of all the forums, another table containing a list of every post made to every forum, another table containing a list of every reply made to every post in every forum.

Now if we go from the start now and say we open the site to show forum1, it first makes a call to the forums list table to get the details of the forum in question. It then makes another call to the posts table to get a copy of all of the posts to do with that specific forum to show on the page. Now on a relatively small database not a big issue but when we are talking large databases (for example needing to search a database table containing many millions of records for the specific records to do with a particular forum it can get intensive.

Now this was a very broad strokes outline of the way that most off the shelf forum software operates. They are designed to cover a reasonably small range of topics that are related in some way, such as a company forum, sports forum (with a forum for a range of the various kinds of sports), etc. At most they may have 10 categories of forums, perhaps 30-40 forums, maybe some sum forums depending on the topic, but this is a far cry from the many thousands that you are proposing. The issue here is that off the shelf forum software is designed to meet the needs of the 90% of cases where a basic average forum software is needed. What you appear to be in need of is a custom developed and engineered forum engine to specifically meet the needs of your site.

You may be needing to look at some of the following...

• Multiple web servers serving your forum engine behind a load balancer to handle high availability and the high load on the system.
• Clustered database server environment running multiple read only replicas to provide a more reliable write service and decrease the load on the main write server.
• A data caching layer and potentially even a page caching layer, to cache the database calls to memory to reduce the amount of database calls being made, and to cache the completed pages to decrease the number of calls to the back end application servers.

These are just some of the things that you would need to look at if you are talking many thousands of forums, and you have to remember to plan ahead when it comes to what your needs are going to be. Each forum may only have 1 post to begin with but each forum could wind up with many many posts in them. Lets crunch some numbers here, there are around 155'100 people employed as actors, producers, and directors in the US alone. If we increase this to say that half the worlds actors, producers, and directors all reside in the US then we can say that there are 310'200 individuals who you would be looking at creating a forum for, and that does not count past individuals who are now deceased. From here lets us give a starting post of one per forum...

At this point we have the following...

• 310'200 records in our forum table
• 310'200 records in our posts table
• 0 questions in our replies table

Now lets say that the forum is in production and people are coming to visit and ask questions and post answers around their favourite actors, producers, and directors. For the sake of argument lets say that an average of 1000 new questions has been asked in the first month alone, and that an average of 500 new questions a month are being asked after that. Then lets say that an average of 10 replies per question is made. Now these are just random values and anyone who has used forum engines previously can probably attest that they are a little on the conservative side. At this point after the first month we have the following...

• 310'200 records in the forums table (this is unlikely to change frequently)
• 310'200'000 records in our posts table
• 3'102'000'000 records in our replies table

And each month after the following number of records are being added...

• 155'100'000 records added to the posts table
• 1'551'000'000 records added to the replies table

So after 12 months in production the database may look something similar to this...

• 310'500 records in the forums table (an extra 300 studios, producers, actors, and directors have come into the business in the last 12 months)
• 2'171'400'000 records in the posts table
• 21'714'000'000 records in the replies table

Now it is true that MySQL can scale quite easily to hundreds of millions of records without exotic hardware requirements but within the first 12 months we are talking many billions of records, with millions of reads and writes per day.

As you can probably see by now doing something like what you are wanting to achieve using a standard off the shelf system is not going to really work out well as it will need to be specially designed to meet your specific use case in order to optimise everything possible for big data.