4

I've created a sitemap index that programmatically generates paginated site maps based on the dynamic content on my site. Google search console has succesfully detected and validated that sitemap when added but does not seem to be following and indexing new pages from the sitemaps.

Here's the relevant info, I think...

  1. My robots.txt approves the sitemap at an external domain, in my case it's a subdomain, but from all I've read this seems to be allowed as long as it's defined in the robots.txt

robots.txt

User-agent: *
Allow: /

Sitemap: https://api.360trivia.com/sitemap?domain=https%3A%2F%2F360trivia.com
  1. Since my site has a large amount of dynamic content, I have opted to generate a sitemap index that paginates said content

sitemap index


<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><sitemap>
        <loc>https://api.360trivia.com/sitemap?resource=questions&amp;page=1&amp;domain=https://360trivia.com</loc>
    </sitemap>
<sitemap>
        <loc>https://api.360trivia.com/sitemap?resource=questions&amp;page=2&amp;domain=https://360trivia.com</loc>
    </sitemap>
<sitemap>
        <loc>https://api.360trivia.com/sitemap?resource=questions&amp;page=3&amp;domain=https://360trivia.com</loc>
    </sitemap>
<sitemap>
        <loc>https://api.360trivia.com/sitemap?resource=questions&amp;page=4&amp;domain=https://360trivia.com</loc>
    </sitemap>
<sitemap>
        <loc>https://api.360trivia.com/sitemap?resource=questions&amp;page=5&amp;domain=https://360trivia.com</loc>
    </sitemap>
<sitemap>
        <loc>https://api.360trivia.com/sitemap?resource=questions&amp;page=6&amp;domain=https://360trivia.com</loc>
    </sitemap>
<sitemap>
        <loc>https://api.360trivia.com/sitemap?resource=questions&amp;page=7&amp;domain=https://360trivia.com</loc>
    </sitemap>
<sitemap>
        <loc>https://api.360trivia.com/sitemap?resource=questions&amp;page=8&amp;domain=https://360trivia.com</loc>
    </sitemap>
<sitemap>
        <loc>https://api.360trivia.com/sitemap?resource=questions&amp;page=9&amp;domain=https://360trivia.com</loc>
    </sitemap>
<sitemap>
        <loc>https://api.360trivia.com/sitemap?resource=questions&amp;page=10&amp;domain=https://360trivia.com</loc>
    </sitemap>
<sitemap>
        <loc>https://api.360trivia.com/sitemap?resource=questions&amp;page=11&amp;domain=https://360trivia.com</loc>
    </sitemap>
<sitemap>
        <loc>https://api.360trivia.com/sitemap?resource=questions&amp;page=12&amp;domain=https://360trivia.com</loc>
    </sitemap>
<sitemap>
        <loc>https://api.360trivia.com/sitemap?resource=blog-posts&amp;page=1&amp;domain=https://360trivia.com</loc>
    </sitemap></sitemapindex>
  1. Each of the linked sitemaps work as I would expect on any standard sitemap setup

sitemap single page


<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"><url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c6e</loc>
        <lastmod>2020-11-24</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c6c</loc>
        <lastmod>2020-11-23</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c6a</loc>
        <lastmod>2020-11-22</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c6b</loc>
        <lastmod>2020-11-21</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c6d</loc>
        <lastmod>2020-11-20</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c55</loc>
        <lastmod>2020-11-19</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c64</loc>
        <lastmod>2020-11-18</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c66</loc>
        <lastmod>2020-11-17</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c65</loc>
        <lastmod>2020-11-16</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c5f</loc>
        <lastmod>2020-11-15</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c5c</loc>
        <lastmod>2020-11-14</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c61</loc>
        <lastmod>2020-11-13</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c5e</loc>
        <lastmod>2020-11-12</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c62</loc>
        <lastmod>2020-11-11</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c63</loc>
        <lastmod>2020-11-10</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c56</loc>
        <lastmod>2020-11-09</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c5d</loc>
        <lastmod>2020-11-08</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c60</loc>
        <lastmod>2020-11-07</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c57</loc>
        <lastmod>2020-11-06</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c5b</loc>
        <lastmod>2020-11-05</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c59</loc>
        <lastmod>2020-11-04</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c52</loc>
        <lastmod>2020-11-03</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c5a</loc>
        <lastmod>2020-11-02</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c58</loc>
        <lastmod>2020-11-01</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c53</loc>
        <lastmod>2020-10-31</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c4e</loc>
        <lastmod>2020-10-30</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c50</loc>
        <lastmod>2020-10-29</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c4d</loc>
        <lastmod>2020-10-28</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c51</loc>
        <lastmod>2020-10-27</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c4f</loc>
        <lastmod>2020-10-26</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c49</loc>
        <lastmod>2020-10-25</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c4a</loc>
        <lastmod>2020-10-24</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c48</loc>
        <lastmod>2020-10-23</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c46</loc>
        <lastmod>2020-10-22</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c3c</loc>
        <lastmod>2020-10-21</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c4b</loc>
        <lastmod>2020-10-20</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad70d27f6004e698c4c</loc>
        <lastmod>2020-10-19</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c47</loc>
        <lastmod>2020-10-18</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c41</loc>
        <lastmod>2020-10-17</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c45</loc>
        <lastmod>2020-10-16</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c3e</loc>
        <lastmod>2020-10-15</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c43</loc>
        <lastmod>2020-10-14</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c42</loc>
        <lastmod>2020-10-13</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c44</loc>
        <lastmod>2020-10-12</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c3d</loc>
        <lastmod>2020-10-11</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c39</loc>
        <lastmod>2020-10-10</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c3a</loc>
        <lastmod>2020-10-09</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c3b</loc>
        <lastmod>2020-10-08</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c40</loc>
        <lastmod>2020-10-07</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c3f</loc>
        <lastmod>2020-10-06</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c37</loc>
        <lastmod>2020-10-05</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c2f</loc>
        <lastmod>2020-10-04</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c38</loc>
        <lastmod>2020-10-03</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c1e</loc>
        <lastmod>2020-10-02</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c36</loc>
        <lastmod>2020-10-01</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c35</loc>
        <lastmod>2020-09-30</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c33</loc>
        <lastmod>2020-09-29</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c34</loc>
        <lastmod>2020-09-28</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c32</loc>
        <lastmod>2020-09-27</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c30</loc>
        <lastmod>2020-09-26</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c31</loc>
        <lastmod>2020-09-25</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c2c</loc>
        <lastmod>2020-09-24</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c2d</loc>
        <lastmod>2020-09-23</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c2e</loc>
        <lastmod>2020-09-22</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c29</loc>
        <lastmod>2020-09-21</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c2b</loc>
        <lastmod>2020-09-20</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c28</loc>
        <lastmod>2020-09-19</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c27</loc>
        <lastmod>2020-09-18</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c2a</loc>
        <lastmod>2020-09-17</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c18</loc>
        <lastmod>2020-09-16</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c1d</loc>
        <lastmod>2020-09-15</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c26</loc>
        <lastmod>2020-09-14</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c24</loc>
        <lastmod>2020-09-13</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c25</loc>
        <lastmod>2020-09-12</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c23</loc>
        <lastmod>2020-09-11</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c22</loc>
        <lastmod>2020-09-10</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c21</loc>
        <lastmod>2020-09-09</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c20</loc>
        <lastmod>2020-09-08</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c1f</loc>
        <lastmod>2020-09-07</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c17</loc>
        <lastmod>2020-09-06</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c1b</loc>
        <lastmod>2020-09-05</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c1a</loc>
        <lastmod>2020-09-04</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c1c</loc>
        <lastmod>2020-09-03</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c16</loc>
        <lastmod>2020-09-02</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url>
<url>
        <loc>https://360trivia.com/q/5f471ad60d27f6004e698c19</loc>
        <lastmod>2020-09-01</lastmod>
        <changefreq>never</changefreq>
        <priority>1.0</priority>
    </url></urlset>

When google search console is setup to hit my main sitemap I get success, but 0 pages or sitemaps indexed. enter image description here

Any ideas why those might not be working as expected?

4
  • 1
    The sitemap is syntactically correct, I've tried several tools and everything comes out ok. My theory would be that you are placing the sitemap in a subdomain, rather than the root domain, and trying to index pages from the main domain. As an experiment, you could try to add your sitemap to the root and request indexing that one. Jan 20 at 18:16
  • Ye after searching some more on that it does seem there may be a limitation for a sitemap to only include hosts from the domain it's listed on. That's the headache for me as my client runs on the main domain (serverless) so I don't have the ability in my current configuration to dynamically generate a sitemap. Going to do some tests and see what turns up. Jan 22 at 17:34
  • I've just written some lambda function to manage rendering my sitemap on the root url and when submitting it now registers all of the sitemaps from my sitemap index. So it seems that having the different subdomain may have been the problem, even though I had it approved in my robots.txt. Still waiting to see if it will fetch the paginated sitemaps, but it's looking promising. Jan 30 at 6:17
  • Looks like it's picking up correctly. @LuisAlbertoBarandiaran if you want to add your theory about the domain not matching as the answer I'll mark it is as the correct answer. Jan 30 at 6:23

1 Answer 1

3
+25

There are a couple of basic things to this:

  1. Sitemaps with cross submits require proven ownership (https://www.sitemaps.org/protocol.html#sitemaps_cross_submits). By default a XML sitemap file can only contain URLs from the same host and from subfolders. To work around this you need to prove ownership by adding a Sitemap: line in the robots.txt file of the other host (https://360trivia.com/robots.txt), which you did. Another way to prove ownership is via verification of the domain property in Google Search Console, however this only works for Google and therefore the Sitemap: line should be preferred. Personally I like to keep my XML sitemaps on the same host as the URLs listed within and within the root directory of that host.
  2. The host of your external sitemaps also needs to allow crawling. The robots.txt for that host shows a 404 (https://api.360trivia.com/robots.txt), which fullfills that requirement. You might want to put a robots.txt file there as well, but for most bots a 404 should work just fine.
  3. Your sitemaps need to be valid XML and the Content-Type header has to state either a text/xml or application/xml, which is also the case.

So from a technical perspective all requirements are met and your XML sitemaps can be used to discover URLs. I verified this for all your sitemaps with our XML-Sitemap Validator. The only problem I see is that the information you give within the XML sitemaps is not very precise and could be optimized:

<url>
<loc>https://360trivia.com/q/5f471aed0d27f6004e6990b8</loc>
<lastmod>2023-11-29</lastmod>
<changefreq>never</changefreq>
<priority>1.0</priority>
</url>

The value of the <lastmod> tag should be in W3C Datetime format. Instead of only the date, you should also add the time and the timezone (e.g. 1997-07-16T19:20+01:00). Having the full information allows search engines to recognize even multiple changes within the same day and increase trust into the data. Unprecise information like <changefreq> and <priority> should be removed, as search engines stated that they ignore this unprecise data anyway.

However, even without changes the sitemap report in Google Search Console should start to show pages once they are crawled and the report is filled with new data, which can take some time and might have a couple of days delay.

6
  • Yes, I would expect to at least see errors as to why those sitemaps are not crawlable, but it seems google hasn't even acknowledged the existence of the sitemaps in the sitemap index Jan 26 at 5:21
  • 1
    What happens if you inspect one of the sitemap URLs in Google Search Console and perform a live analysis? Uncrawled does not mean it is uncrawlable, sometimes it just takes a while. You can also consider adding a proper lastmod date for the sitemap index file with the lastmod date beeing the newest date of all URLs within. Another useful thing is to have an additional "hot sitemap" that only contains the last (e.g. 1000) changed URLs. That file will change more frequently and that will cause a higher crawl frequency as well, leading to faster change discovery. Jan 26 at 5:34
  • That's ^ a good idea. I tried putting in just one of the sitemaps and got Sitemap could not be read. Looking into this some more. Jan 30 at 1:14
  • Regarding <lastmod>, walmart.com/sitemap_auto_dir.xml doesn't seem to be an issue using date only format. Jan 30 at 1:23
  • Only using a date is not really an issue, but also not ideal. Just consider an online shop with multiple price updates per day, or a news website with updates or error corrections. If you only provide the data, there is no way to tell URLs need to be recrawled. Jan 30 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.