2

I have a site where I use robots.txt to reference my sitemap, since it's not located under the default /sitemap.xml.

However if I go to google webmaster tools, I cannot use the sitemap debugging tool to debug it, since it's not submitted through that tool (according to google). The reason for this is, that I use robots.txt to reference it, since the the sitemap is hosted on S3.

Now I would like to verify, that google reads and understands it - how can I do that?

  • 4
    Why not submit it through Google Search Console?? That solves everything. – closetnoc Jan 4 '17 at 16:17
  • 1
    @closetnoc I think OP can't submit it through search console because it is hosted on s3(like https://s3.amazonaws.com/userxyz/sitemap.xml/, but in search console webmaster can submit the sitemap within a site directory or relative URL like /sitemap.xml or /some-folder/sitemap.xml, webmaster have not choice to use absolutely URL. – Goyllo Jan 4 '17 at 18:48
  • 1
    @Goyllo Gotcha! Then the OP may have another problem. The sitemap must be on the domain for which the sitemap URLs are for. If the OP can access the sitemap as OPdomainname.com/sitemap.xml then that should work okay. Good thought! Cheers!! – closetnoc Jan 4 '17 at 19:26
2

You'll need to submit the XML sitemap in Google Search Console (formerly Google Webmaster Tools) in order to verify this.

It's quite possible that Google is already picking up your sitemap, having seen the reference in robots.txt. (Since Google does support the Sitemap directive in robots.txt) You should be able to check your access logs to determine whether Googlebot has requested it. However, that doesn't tell you whether Google found any problems with it.

I tried this some time ago myself. With the XML sitemap referenced only in robots.txt, even with the sitemap in its "default" location ie. example.com/sitemap.xml, there is no acknowledgement from Google unless you explicitly add it to GSC.

The Sitemap: directive in robots.txt is supported by all major search engines, not just Google.

It's not necessary to submit the sitemap in GSC for Google to pick it up, but it is necessary if you want to report on and diagnose it.

The reason for this is, that I use robots.txt to reference it, since the the sitemap is hosted on S3.

However, whilst the Sitemap URL can be on a different host to the robots.txt file, if the URLs in the sitemap are referencing a different URL-path to where the sitemap is located then this could be a problem. From Sitemaps.org:

...all URLs listed in the Sitemap must use the same protocol ... and reside on the same host as the Sitemap. For instance, if the Sitemap is located at http://www.example.com/sitemap.xml, it can't include URLs from http://subdomain.example.com.

You might be able to get around this with Google by registering all properties in GSC, but can you verify S3? And this won't work with other search engines.

  • That's right search engine like ask.com still follow the sitemap from robots.txt – Goyllo Jan 4 '17 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.