1

Is there a way to test multiple sitemap.xml file? Validation works fine, Google is accepting all sub files, but "Server response check" in Yandex is returning "Document contains no text".

By the crawl rate and overall indexation progress I get the impression that both search engines fail to read content from the sitemap files. Both because there is a large amount of "Discovered – currently not indexed" = 2/3 of all content, they have never been crawled and because of low ration indexation in Yandex.

This website includes about 750.000 links in sitemap files. When I generate 50.000 links per file (about 11mb), crawling graph goes high and then drops. When it's 10.000 links per file, graph falls much faster and stays on about same level further on.

We did various checks and technically all seems right, but looking at performance it's quite doubtful. Robots.txt gives full access. robots meta tags also.

  • Can anyone suggest a way to check why "Server response check" would return an error when the file is there?
  • Is there a way to test if the whole system of sitemap files really works - meaning actually does being read properly by the search engines?
  • Can this issue be related to settings set in .htaccess file?

Please see screenshots below. Sitemap file location: https://www.rusinfo.eu/sitemap.xml Yandex Server Check link: https://webmaster.yandex.ru/tools/server-response/

Thanks in advance enter image description here enter image description here enter image description here enter image description here enter image description here enter image description here

Improve this question
4
  • You can't just put URLs in a sitemap and expect search engines to index them and rank them well, that just doesn't happen. See The Sitemap Paradox. Search engines will index some URLs from sitemaps, but if you want more of your URLs to get indexed, or if you want them to rank well, you need to link to them on your site.
    – Stephen Ostermiller ♦
    Aug 29, 2019 at 12:51
  • Hi Stephen, I am not much worried whether search engines will include all links but rather that it will function properly. 2/3 of all pages have never been crawled. Considering the peaks vs sitemap size it might be that this is a result of not being able to find these pages, meaning sitemap is not informing SE about all existing pages. I would say let it be, but Yandex confirms that with errors while opening sitemaps
    – igi
    Aug 29, 2019 at 16:04
  • Sitemaps are not a good way of informing search engines about pages. It is much better to link to the pages from other pages. You should just forget about sitemaps, for the most part they are a big red herring.
    – Stephen Ostermiller ♦
    Aug 29, 2019 at 16:11
  • What do you mean with "better to link to the pages from other pages"? Each post includes link to latest 100 news posts (or viewed etc, depending on filter set). As for informing search engine, we have just recently implemented a built-in pinging system, which pings hourly Google in this case to check the updates. It functions just for a few days but there was no impact so far. Sitemap is a very important tool when it comes to websites with high volume of content, so I would really like to find the solution to make Yandex access the files and let Google at least crawl/notice all pages
    – igi
    Aug 29, 2019 at 16:20

1 Answer 1

Reset to default
1

"Server response check" in Yandex is returning "Document contains no text".

enter image description here

enter image description here

if the whole system of sitemap files really works - meaning actually does being read properly by the search engines?

Sure: error-free log-files and alert-free GSC/Yandex Webmaster would mean, there is everything ok with sitemaps.

Can this issue be related

What issue? Do you mean

This website includes about 750.000 links in sitemap files. Indexed are about 150.000

That is because your website has too few unique content. Most content is duplicated - published firstly by other media. For Google are most parts of the site just secondary news aggregator.

What you experience isn't a sitemap issue, but massive duplicated content issue.

Improve this answer
2
  • See What is duplicate content and how can I avoid being penalized for it on my site?
    – Stephen Ostermiller ♦
    Aug 29, 2019 at 12:52
  • Hi Evgeniy, As I mentioned in the post, Validation works fine, but Server Response check returns a blank document ("Document contains no text"). Unfortunately it's not possible to add pics in replies, so I added the screenshot to the original post. webmaster.yandex.ru/tools/server-response If you look at the 2nd screenshot (crawl rates), the last peak is low (= 10.000 links), the peak before is much higher (= it was 50.000 links). Server Response result + errors accessing sitemap files (in screenshots 5-6) seem as clear indication that yandex reads only the first sitemap
    – igi
    Aug 29, 2019 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.