GitHub allows you to create a custom 404 page but there is a caveat. The 404 will be displayed only if you use a custom domain or for user pages.
Custom 404s only work when they are served from the root of a Pages domain, for example a project using a custom domain or a user pages repository. A custom 404 page on project pages that are not using a custom domain will not work
In other words, the 404 error page will not work for project pages with no custom domain.
In your case, it looks like you are using a custom domain thus the limitation may not apply to you. Are you sure your 404 is working correctly and you don't have an issue elsewhere?
Just to show you an example, in our company blog we use Jekyll, we host the page on GitHub pages and we have a custom 404 error page. As you can see, it also works on a subfolder.
Try to visit this page
and you can check yourself. If it helps you to debug, the pages are generated directly by GitHub. We have another repository where we publish the HTML pages to GitHub (instead of a Jekyll powered website), I should check if in that case it also works.