Well the correct way in your case (which is: the user consciously requests an invalid resource) is to return a 404-response code. The body of the response is a description for the user of your site. If your 404.php is just like a template and contains mostly HTML-code then you could just include it. Otherwise showing the "project '$name' does not exist" is the best alternative.
So as an example, in PHP it would look like this (of course if you use a modern PHP framework then you can just use some kind of NotFound exception type):
if(project_exists($name)) {
// show the project like you do now
} else {
// project not found, return a 404 respose code
header('HTTP/1.0 404 Not Found');
// generate the response body either by
// including your 404-page (if it's mostly HTML, like a template)
include('/path/to/404.php');
// or just stating that the resource was not found
echo 'Project "', $name, '" does not exist!';
}
Regarding your options:
Number 1 doesn't make much sense - if a resource has never existed, then there is no point in redirecting (code 301 or 302) it to the home page or even a 404 page.
This only makes sense if the resource was deleted and you know what the new path is (for example if you have a map with old and new paths/project names) or if you can guess what the user wanted and redirect him to the proper resource (for example you could "fix" mistyped URLs with a fuzzy search). Of course you could also solve that with a "smart" 404 page.
Number 2 won't work. If I understand you correctly you would return a 404 response code and add a Location-header. But 4xx-codes must not return Location headers - only 3xx and 201 (and 202) response codes allow the Location header.
Even from a SEO perspective you should just use 404. First, there should be no links to invalid resources anyway and if you do something fancy here you hurt the people who actually use a link checker to check for invalid links on their side.
If the address of an existing resource changes, then it makes sense to use 301.
If the existing resource is just deleted (as in 410 Gone), then a redirect to a generic site (like the home page) may keep google happy, but it might not be user friendly (I hate it when I read an article about $TOPIC, click on a link in that article and get redirected to a total different page b.c. the link is wrong). You could use some light grey-hat stuff here like just redirect the Google/Bing/Yahoo bot and show 404 to the rest.
