One way to do it is to write a
FileBackend subclass for using the DB as a backend. There is not much documentation on how to do it, though. (This would assume that you still store the names of files in the wiki DB, and only use your custom DB to store file contents. If you want to delegate all file handling responsibility to the custom DB, you'll also need your own file repo class as well. Also not well documented; see
$wgForeignFileRepos as a starting point.)
A less elegant but easier approach is to write a tag extension which takes an image name/id and outputs the HTML for the image, maybe as a
data: URI (or using a special page as the
src and doing the DB lookup there). You'll have to reimplement all the image handling (HTTP headers, HTML formatting etc) that MediaWiki would otherwise do for you, though.