The distinction comes from how registration is done between registrars and registries, using the EPP protocol, and does not come from the DNS plane.
On the resolution side of things, there is no distinction. Each zone is served by a set of nameservers. This is all that exist. But we can already introduce here terminology that would be useful:
- nameserver A is said to be "internal" for the zone Z if its name is inside the same TLD (or more precisely public suffix);
ns.foobar.example being authoritative for name gloubiboulga.example is internal to the example TLD,
- nameserver A is said to be "in-bailiwick" for the zone Z if it (its name) is inside the zone (or even the same as the zone name, but this is a case best to be avoided);
ns.foobar.example being authoritative for foobar.example is in-bailiwick because it resides in the zone it serves.
Now these distinctions are not purely theoretical as they have technical consequences: in-bailiwick, and sometimes internal nameservers need "glue" records at registry. Which means that the registry does not only publish the NS record for the zone delegating it to those nameservers as it happens for any domain, it also has to publish A/AAAA records, otherwise the resolution won't happen for an obvious chicken/egg problem. Remember this specific point on IP addresses in specific cases later.
Let us go back to the registration plane, so what happens between registries and registrars, and by extension to end clients. Most of it is nowadays governed by a model coming from EPP, being the protocol used by registrars to connect to registries and conduct their operations.
While not the only way of doing things, EPP defines various "objects" that resides on the registry side and on which registrars act. There are domains, of course, but also contacts, and also "hosts". A host is a nameserver technically, but just in the registry database (or more precisely in the EPP model) it is called a host.
How it is expected to happen? When you, through a registrar, says that domain X should be updated to use nameserver Y, the registrar has this to say:
- check if nameserver Y exists as host object in the registry database, and if not, create it if possible;
- then, and only then, can the registrar update the domain to link the host object to the domain.
A registrar can create host objects only under the zones it sponsors at registry, or external (the opposite of internal) nameservers. If registrar A maintains example.com in the .com registry, it can create any nameserver (host object) below example.com, such as ns1.example.com, or any external nameserver, such as ns.elsewhere.example if .com and .example are not in fact handled by the same registry. But if registrar A tries to create host object ns1.example.com where domain example.com is in fact sponsored by registrar B, this will be denied by registry.
Which explains why above the registrar can create the needed host object only in some cases.
But this also explains why a registrar can also create host objects that are not used for domains it sponsors, and why it has to allow its customers to do so.
Let us take one example.
- domain
bigprovider.example is sponsored by registrar A; let us say it uses itself, as authoritative for this domain name, nameservers ns1.example.com, ns2.example.com, and so on (so they are all external in fact)
- but this provider does provide DNS service for its customers, hence it tells its customers to set up their domains with
ns1.bigprovider.example and ns2.bigprovider.example
- so customer Foobar can register its domain
foobar.example through any registrar, let us say registrar B, and to associate this domain to nameservers ns1.bigprovider.example and ns2.bigprovider.example these 2 nameservers must exist at registry as host object, and this can be done only through registrar A, and even if those hosts are not used as authoritative for any domain sponsored by registrar A.
So this should explain the two options you describe.
Now let us go back to the issue of IP addresses. As we said, you need (but it can be done transparently by your registrar in some/most cases, see above) to create the nameservers as host objects before being able to use (link) them for any domain. So to create those host objects you need to provide a name (obviously) but also you may provider one or more IPv4 or IPv6 addresses.
Why? Because if the server is in-bailiwick or internal, it needs to have IP addresses specified. If example.com uses ns1.example.com as authoritative, the registry authoritative nameservers need not only to give back DNS record example.com. NS ns1.example.com. as for any domain to show the delegation but will also need to provide so called glue records such as ns1.example.com. A 192.0.2.42 otherwise the DNS client would never be able to find really where to connect to, to finally reach the delegated nameservers.
As a registrar can (sometimes) change the name of a host object, it means something internal can become external or the reverse, and something in-bailiwick can become out of bailiwick or the reverse. If you followed closely the above, this all means that some host without IP addresses may suddenly need some or in the opposite direction, some existing IP addresses may become obsolete. The rules of host objects creation per registry varies. Some will let you enter IP address when not needed (for example if the host is external anyway), they are just stored in the database and will be used maybe in the future if the host changes names. Some others won't allow that.
Also, for anything related to terminology, I recommend you read and stick to RFC 8499 DNS Terminology; it has a long discussion of all the above in-bailiwick/internal thing at §7 about Zones.
Do remember however though that this document is purely technical and won't necessarily match exactly what you find elsewhere, especially when people are not strict on their terminology or start to use more marketing terms often not precisely defined.
PS: as for "EDIT: I noticed I can't add in the hostname field anything with a dot:" this just means your DNS provider is adding a specific local constraint that is absolutely not needed; either they have some valid reason not given, or they don't understand the DNS too much in fact (which is often the case outside of big DNS provider, this is a job by itself, even if registrars often do it, this is still completely different from providing domain name registration services, which is the job of registrars). If you have domain example.com, it is 100% correct and possible to create a host object of a.ns.example.com or any other "complex" name and use it; multiple providers do this in fact, for example if you use CloudFlare as DNS provider they will tell you to associate your domain name to nameservers having a name akin to jules.ns.cloudflare.com and so on.
Appendix: On the EPP model and host vs nameserver.
If you look at RFC5731 which defines domain name operations through EPP, you will see for example when fetching information on a domain a reply like:
S: <domain:ns>
S: <domain:hostObj>ns1.example.com</domain:hostObj>
S: <domain:hostObj>ns1.example.net</domain:hostObj>
S: </domain:ns>
S: <domain:host>ns1.example.com</domain:host>
S: <domain:host>ns2.example.com</domain:host>
You can see here the distinction between ns (which gives the authoritative hostnames currently attached to this domain name) and host (which gives any host object existing at the registry under this domain name).
The RFC text on those elements is the following:
An OPTIONAL domain:ns element that contains the fully qualified
names of the delegated host objects or host attributes (name
servers) associated with the domain object. See Section 1.1 for a
description of the elements used to specify host objects or host
attributes.
Zero or more OPTIONAL domain:host elements that contain the
fully qualified names of the subordinate host objects that exist
under this superordinate domain object.
