On our website we noticed that the Applebot is not crawling our sitemap.xml, so it is unaware of most of our internal webpages.
We have no robots.txt restrictions for that bot, on the contrary we tried to whitelist it, and there is a Sitemap field containing the sitemap URL as you can see below.
Are we doing anything wrong in robots.txt which is blocking it from crawling our sitemap.xml?
Here's our robots.txt looks like, with minor changes such as replacing our domain with example.com:
## Disallow ALL
User-agent: *
Disallow: /
Crawl-delay: 600
## but allow only important bots
User-agent: Applebot
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Mediapartners-Google
User-agent: msnbot
User-agent: msnbot-media
User-agent: Slurp
User-agent: Yahoo-Blogs
User-agent: Yahoo-MMCrawler
User-agent: Yandex
## disallow directories
Disallow: /dir1/
Disallow: /dir2/
## disallow files
Disallow: /status
Disallow: /health
## disallow some text file extensions
Disallow: /*.txt$
Disallow: /*.json$
Sitemap: https://example.com/sitemap.xml
Host: example.com