I am using a custom domain for a GitHub Pages site. They said to set
A record. But what nameservers should I use? What is the default, or should I leave it blank?
You answered your own question.
As for the Nameservers, unless GitHub says otherwise, you are going to ask what the A Record or CNAME should be set on your domain name. Then, if you don't know how, ask your Domain Register about setting the CAME or A Record to what it needs to be for your domain name.
AFAIK unless somethings changed you arent able to use Namesevers as they are used for the main domain, you will have to use a A domain or a CNAME to cover the domain. Sorry to break the news to you.