5

One can sort robots.txt this way:

User-agent: DESIRED_INPUT
Sitemap: https://example.com/sitemap-index.xml
Disallow: /

instead:

User-agent: DESIRED_INPUT
Disallow: /
Sitemap: https://example.com/sitemap-index.xml

I assume both are okay because it's likely the file is compiled in correct order by generally all crawlers.
Is it a best practice to put Disallow: before Sitemap: to prevent an extremely unlikely bug of a crawler's bad compilation of crawling before ignoring Disallow:?

1 Answer 1

2

In this case, it doesn't matter. I would list the sitemap URL at the very bottom of the robots.txt, to keep things from being randomly jumbled up (it makes no sense to have it in the middle of a directive), but technically, this won't make a difference. You just want to have things properly ordered for yourself, and anyone else working on the website.

In your first example, the crawler will see its name called out, note the sitemap URL, crawl it at some point as it crawls your site, and meanwhile continue down the list of directives.

Will it matter if, instead of User-agent: *, you wrote User-agent: Googlebot? Will that only point out the sitemap to Google and not Bing or other crawlers? If your sitemap is at the root of your domain and not otherwise blocked or unavailable, search engines will find it anyway as they crawl your site. In fact, the sitemap directive is optional, more of a traditional thing that SEO's do. (Unless you stuck it in a directory that's not obvious to crawlers, in which case, you should move it to the root if possible, anyway.) Of course, if the point is to explicitly point out your sitemap to all crawlers, putting it on the bottom, or at the top for that matter, will make more sense.

What does matter (somewhat - not for the major bots, per MrWhite's notes below) is where you put Allow vs Disallow. Let's say you have the following:

User-agent: *
Allow: /category/stuff
Disallow: /category/

In this case, the best practice is to put Allow first, so that there's no chance that a non-mainstream crawler will see a blocked directory and won't bother checking the single allowed file within.

3
  • 1
    "What does matter is where you put Allow vs Disallow ... the best practice is to put Allow first" - For maximum compatibility; yes. Although strictly speaking it should be the most specific rule first, regardless of whether it is Allow or Disallow - although Allow directives are expected to be more specific, so they would naturally go first.
    – MrWhite
    Commented Sep 16, 2019 at 12:15
  • 1
    However, all mainstream search engine bots (Googlebot, Bingbot, YandexBot, ...) match according to the longest (most specific) path (which is in accordance with the recently proposed spec - "Robots Exclusion Protocol"), regardless of the order of the rules. So, for the mainstream search bots, it doesn't actually matter where you put the Allow and Disallow directives.
    – MrWhite
    Commented Sep 16, 2019 at 12:18
  • Thank you for the notes and the spec link, @MrWhite -- makes sense. I modified my answer to include a reference to your updates. Commented Sep 16, 2019 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.