Skip to main content
2 of 2
added 76 characters in body
Henry Visotski
  • 5.2k
  • 10
  • 18

In this case, it doesn't matter. I would list the sitemap URL at the very bottom of the robots.txt, to keep things from being randomly jumbled up (it makes no sense to have it in the middle of a directive), but technically, this won't make a difference. You just want to have things properly ordered for yourself, and anyone else working on the website.

In your first example, the crawler will see its name called out, note the sitemap URL, crawl it at some point as it crawls your site, and meanwhile continue down the list of directives.

Will it matter if, instead of User-agent: *, you wrote User-agent: Googlebot? Will that only point out the sitemap to Google and not Bing or other crawlers? If your sitemap is at the root of your domain and not otherwise blocked or unavailable, search engines will find it anyway as they crawl your site. In fact, the sitemap directive is optional, more of a traditional thing that SEO's do. (Unless you stuck it in a directory that's not obvious to crawlers, in which case, you should move it to the root if possible, anyway.) Of course, if the point is to explicitly point out your sitemap to all crawlers, putting it on the bottom, or at the top for that matter, will make more sense.

What does matter (somewhat - not for the major bots, per MrWhite's notes below) is where you put Allow vs Disallow. Let's say you have the following:

User-agent: *
Allow: /category/stuff
Disallow: /category/

In this case, the best practice is to put Allow first, so that there's no chance that a non-mainstream crawler will see a blocked directory and won't bother checking the single allowed file within.

Henry Visotski
  • 5.2k
  • 10
  • 18