1

On our website we noticed that the Applebot is not crawling our sitemap.xml, so it is unaware of most of our internal webpages.

We have no robots.txt restrictions for that bot, on the contrary we tried to whitelist it, and there is a Sitemap field containing the sitemap URL as you can see below.

Are we doing anything wrong in robots.txt which is blocking it from crawling our sitemap.xml?

Here's our robots.txt looks like, with minor changes such as replacing our domain with example.com:

## Disallow ALL
User-agent: *
Disallow: /
Crawl-delay: 600

## but allow only important bots
User-agent: Applebot
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Mediapartners-Google
User-agent: msnbot
User-agent: msnbot-media
User-agent: Slurp
User-agent: Yahoo-Blogs
User-agent: Yahoo-MMCrawler
User-agent: Yandex

## disallow directories
Disallow: /dir1/
Disallow: /dir2/
## disallow files
Disallow: /status
Disallow: /health

## disallow some text file extensions
Disallow: /*.txt$
Disallow: /*.json$

Sitemap: https://example.com/sitemap.xml
Host: example.com
4

3 Answers 3

1

A single typo looks to be causing this issue.

Your disallow: / statement is blocking all robots. It should show up as disallow:

Here is a rewritten version of your robots.txt file to allow robots to access your sitemap...

User-Agent: *
Disallow: 
Disallow: /dir1
Disallow: /dir2
Disallow: /status
Disallow: /health

Sitemap: https://example.com/sitemap.xml
0

@PlanetScale

Since the default for URLs is 'index' (for Google, at least), why leave a blank line in robots.txt with regards to Disallow:?

I'd omit it before I'd specify it.

-1

Keep these lines at last.

## Disallow ALL
User-agent: *
Disallow: /
Crawl-delay: 600
2
  • Surely you jest. You are recommending that they disallow all user agents, to the home directory and below with a 600ms delay. If you are disallowing all user agents from the domain home level and depper, why would a crawl delay matter?l? This is why you shouldn't outsource SEO.
    – adam-asdf
    Feb 13, 2017 at 9:44
  • @adam-asdf. Thank you for your pointer. I am new to SEO. I do not take outsourced projects and neither I am planning to take in the future. May be my answer is wrong. You should stop generalizing.
    – Mayur
    Feb 13, 2017 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.