On our website we noticed that the Applebot is not crawling our sitemap.xml, so it is unaware of most of our internal webpages.
We have no robots.txt restrictions for that bot, on the contrary we tried to whitelist it, and there is a Sitemap field containing the sitemap URL as you can see below.
Are we doing anything wrong in robots.txt which is blocking it from crawling our sitemap.xml?
Here's our robots.txt looks like, with minor changes such as replacing our domain with example.com:
## Disallow ALL User-agent: * Disallow: / Crawl-delay: 600 ## but allow only important bots User-agent: Applebot User-agent: Googlebot User-agent: Googlebot-Image User-agent: Mediapartners-Google User-agent: msnbot User-agent: msnbot-media User-agent: Slurp User-agent: Yahoo-Blogs User-agent: Yahoo-MMCrawler User-agent: Yandex ## disallow directories Disallow: /dir1/ Disallow: /dir2/ ## disallow files Disallow: /status Disallow: /health ## disallow some text file extensions Disallow: /*.txt$ Disallow: /*.json$ Sitemap: https://example.com/sitemap.xml Host: example.com