Our application is growing fast so now we need to scale the infrastructure to don't slow down the entire website independently on how many users we have

The backend system is based on many curl calls lasting 1-10 seconds and they need to be executed in parallel

We currently have a 4gb ram 4 core vps but sometime we get 503 error when all the 50 channels start in parallel (we setup 50 cron job) that execute 5 curl call/each

These numbers are going to increment fast so we need to find a solution to don't let the end user wait more than a minute to get his things done

Do we need a dedicated server, or it won't make much difference? Or set up aws lambda functions with sns queue? Amazon would be the best solution because whatever amount of sns messages we put in the queue and instantly they will get processed by enabling the trigger, but the lambda function currently don't support PHP so we would to rewrite all the code in Python

How would you manage this situation to improve the performance?

Thank you


This sound like you need to improve your code instead of your hardware. You don't provide an example in your question so I cant assess the quality, so I've based it on the situation described.

Those cronjobs, why does that need to be more when you get more visitors? To me, that is a small red flag. I should not have to maintain my code per visitor.

E.g.1: Do you fetch something for the user? If so, can't you make a [click here to get result] so that you only cURL when the user clicks? That'll save you a lot of resource AND it scales easier because only active users require resource.
E.g.2: Do those cronjobs have to start at the same time, or can you chain them? I have a script which is quite intense, it loops through all products of all brands. But, I can afford some delay, so every 25 products it sleeps 1 second, after every brand it sleeps 10seconds. This dropped the 30% by spreading it across a longer period of time.
E.g.3: Is it relevant for all code to be updated all the time, or can you move this in the nightly hours? This allows you to push the CPU more as you don't ennoy users and there are no users to slow down the process.

This might not applicable to your situation, but you might want to try thinking of more examples. Too often people just up their hardware instead of improving their code. Might even be cheaper in the long run because you don't need to get a more expensive server.

If all of this isn't applicable and your code it of high quality performance wise, you might want to check out VPS services. Those often allow expention with a few clicks, e.g. you add another core or memory via your controlepanel and turn it off after your spike.

  • Thank you. The jobs need to scale because each user put from 10 to 250+ jobs at once (each one doing one curl call) and they need to be executed all within a minute otherwise we'll slow down their work. It's rare that we get more than one user putting these jobs at the same time but if it happens and they together put like 1000 jobs the second one will have to wait up to 5 minute because our system currently process only 250 in a minute (50 channels x 5 curl each) and we need a solution to increase this number to unlimited – lorigio Feb 21 '17 at 10:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.