3

I created a sitemap (which is generated dynamically) and routed to it using reverse-proxy in Nginx. I uploaded the URL to the Google search console. I got an error right away as seen in the picture. "Sitemap is HTML"

screenshot of Google Search Console with the Message "Sitemap is HTML"

After digging around a while it appeared our pre-renderer had picked up the request and served to google a pre-rendered version of the XML file, thus in Html.
But even after fixing this, making sure no request for sitemap.xml goes to our pre-renderer google still, gives the same error message.

I have tried removing it and adding it again in the search console multiple times on different days, I have tried waiting, I have tried serving it with another name (sitemap2.xml), and I have tried adding an actual XML file instead of the dynamic one. Nothing works!

I have verified the XML file after I disabled the pre-renderer on multiple other sites and everyone gives me an "ok". It's as though it's ignoring my requests to re-check the file.

Sitemap location: https://www.tirex.se/sitemap.xml

Edit 1:
My current status as of 2021-09-02: Latest update

Improve this question
3
  • 3
    How long has it been since you fixed the problem? You said you tried waiting, but you didn't say how long. It usually takes Google a couple weeks to update. Have you tried inspecting the sitemap URL and doing a "live test?"
    – Stephen Ostermiller ♦
    Aug 30, 2021 at 12:17
  • 1
    @StephenOstermiller: I have been waiting since the 23:e of august, so a week. It could take multiple weeks? :O Yes, I tried inspecing the sitemap manually, in every way I could think of. Including visiting with different browser and using googlebot as user-agent. Also requesting other sites that I found to validate the XML.
    – Wirde
    Aug 30, 2021 at 12:20
  • 2
    Google usually takes two weeks to update pretty much anything. I wouldn't be surprised if it took 4 to 6 weeks.
    – Stephen Ostermiller ♦
    Aug 31, 2021 at 1:23

1 Answer 1

Reset to default
3

I checked the sitemap you posted with the network tab open in dev tools and I'm getting a 304 in the request headers.

Edit: I did clear cache and refresh while I was answering and was still seeing 304. I just checked again and can confirm that I am no longer. Can't explain that unless something changed.

If for some reason Google got a 304 at some point they'd be redirected to a previously (locally) cached result, which in your case could ostensibly be the old pre-rendered version (html).

Make sure that https://example.com/sitemap.xml returns 200 and to be safe double check your validation/fixes.

You also mentioned that you tried renaming the file - to further troubleshoot I would block the old version via robots.txt and also in that file list your sitemap.

User-agent: *
Disallow: 
/sitemap2.xml

Sitemap: https://example.com/sitemap.xml

Stephen is right about the time it takes to update, typically does take a couple weeks.

Edit2: Considering that you're using etags, I would recommend using an RSS/Atom feed in addition to your XML sitemap. RSS/Atom feeds might be more reliable, and will provide all updates on your site and keep your content fresher in Google's index.

Improve this answer
13
  • 3
    If you clear your cache, you shouldn't get a 304. Their server uses etags, so it should give a 304 if you have a copy, you refresh, and it hasn't changed. Seems to work fine with curl: $ curl --head https://www.tirex.se/sitemap.xml: HTTP/2 200
    – Stephen Ostermiller ♦
    Aug 31, 2021 at 1:29
  • 1
    I tried clearing caches and refreshing while I was answering and was still getting 304. Checked with Chrome and Brave. Just tried curling and confirm 200 and re-checked same processes as before and now getting all 200 without clearing/refreshing.
    – Mike Ciffone
    Aug 31, 2021 at 1:57
  • 1
    @StephenOstermiller Could it be possible that If-None-Match failed for some reason? That would cause the server must return HTTP status code 304, no? IDK I'm not confident/comfortable with it.
    – Mike Ciffone
    Aug 31, 2021 at 2:24
  • 1
    Would it be wise to turn of etags for the sitemap request? It could very likely be that the server is returning 304 to google considering that we havn't made any changes to the sitemap since first upload. (That I'm aware of, it's dynamicly generated, so maybe...) @MikeCiffone: How would RSS/Atom benefit this situation? Why might it be more reliable? And what should be in it? Updates to the sitemap? Sorry I don't get it :P
    – Wirde
    Aug 31, 2021 at 8:44
  • 1
    @Wirde In general many recommend submitting both rss/atom + sitemap: one for recent changes and the other for your full "sitemap". In this case, if you include all of your pages, it would be a different form of sitemap (that Google for sure wouldn't have cached unless you already had it) that you could submit in Search Console to get what you care about done quicker (your content indexed?). If you want to keep bandwidth down, be specific about what bots you allow (but for sure googlebot).
    – Mike Ciffone
    Sep 1, 2021 at 3:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.