I am using a custom domain for a GitHub Pages site. They said to set
A record. But what nameservers should I use? What is the default, or should I leave it blank?
You answered your own question.
They said to set CNAME or A record.
As for the Name servers, unless GitHub says otherwise, you are going to ask what the A record or CNAME should be set on your domain name. Then, if you don't know how, ask your Domain Name Registrar about setting the CNAME or A record to what it needs to be for your domain name.
The "default" nameservers are typically the nameservers owned by your domain registrar, and your registrar should have a button to fill them in automatically. If edits to your
CNAME records are working, that means your nameservers are right already and you don't need to make any modifications to them.