I hope this is in the right topic area. Sorry if it isn't.

I want to ask the right way to do this task.

I have a few service API's I need to query every 30seconds or so that I would like to store in a mysql table then use this data on a web page.

Now I have done this using a python script to get all the data, format it and update the table and this works fine. Now at the moment I'm using a cron task to run the script every 30 seconds.

My question is: - is cron the right way to run this script in intervals or am I going against best practice? - Am I going about this whole thing completely the wrong way to get API data?

I was thinking about having the script trigger at the bottom of the HTML code, but if multiple people access the page it will trigger the script each time ending in tens if not 100's of python scripts querying the APIs

PS: I have done a search but couldn't see a post that answers this question.

Cheers in advance :-)

  • 3
    What makes you think cron is wrong? If you need your script to run reliably periodically (eg. every N units of time), then you are going to need a task scheduler, like cron.
    – MrWhite
    Commented May 17, 2015 at 22:52
  • I use cron for all my scheduled automated data collection.
    – closetnoc
    Commented May 17, 2015 at 23:40
  • OK so cron is how this task is achieved. I didn't know if there was a security issue or other, running these things in cron. As long as I follow best practice using cron I should be right then. I just wanted to know how web developers do this, just so I'm not opening up myself to issues in the future. Thanks heaps guys!! :-) Commented May 18, 2015 at 0:01
  • Sounds like the most efficient answer requires programming. I think this question is better suited for stackoverflow.com Commented May 18, 2015 at 2:29
  • Well kind of, I have no issue with the programming side of things. I was wanting to know if I was to hire a web development company to build the site I'm building would they run a script in cron to gather data or is there a "proper" way. From the comments above it looks like I'm doing it the proper way. Cheers guys :-) Commented May 19, 2015 at 21:30

1 Answer 1


In this instance you have taken the option that is the best practice and is what is done by most websites that have to run tasks at regular intervals. What it boils down to is you are using the servers native task scheduling tool (cron) to run a regularly scheduled task, this is absolutely what you need to do. Running it every single time the page loads not only will cause a huge number of queries to be made to the API every single second, but will also slow down the page load as the user will have to wait while the server side code checks the API's for data and commits the data to the database. What you have done achieves everything you need, uses tools for what they where designed to do, and does not cause issues for your end users' experiences. Well done.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.