Only allow crawling of pages in sitemap

Question

My idea is, to use robots.txt to disallow everyting and put all allowed pages into the sitemaps.

User-agent: *
Disallow: /
Sitemap: https://www.example.com/sitemap.xml

Is this working as expected?

Answer 1

No, that won't work. That disallows all crawling. The sitemap: directive in robots.txt says where the sitemap is, but has no bearing on whether or not the URLs in it are crawlable. With this setup Google Search Console will report errors in your sitemap:

Sitemap contains URLs which are blocked by robots.txt.

I wouldn't recommend trying allow only crawling of URLs in your sitemap. Technically it could be possible with something like:

User-agent: *
Disallow: /
Allow: /$
Allow: /some/page.html$
Allow: /other/things.html$

where you list all the URLs that are in your sitemap ending in $ to make the allow rules exact rather than "starts with". However robots.txt is not designed to get very large and it will soon exceed the size supported by many robots. In addition, most bots don't support the Allow: directive. For most bots, it would completely disallow all crawling. Only the major search engines which support Allow: would be able to crawl anything.

I usually recommend allowing crawling as much as possible and then using URL canonicalization and noindex tags to control what gets indexed. robots.txt Disallow: is most useful when crawling itself is problematic; either because there are too many URLs or because crawlers cause unwanted side effects such as messing up stats.