For a small business in this case, this is extreemly practical. I don't think the site be doing any of the following:
1.Accepting payements
2.Being viewed by more than 10 people in a day
3.having lots and lots of data being updated, maintained etc.

You can get a crappy old computer and install LAMP on it, leave it connected on and forget about it.

This way you will have full control of everything while you are still developing etc. 

You can move the files etc. to a webhost later if it is needed. You will also learn more about website maintenance issues by trying it yourself.

Just keep a back up copy of files / databases needed to make the site, but you have to do the same even if you were hosting somewhere else.