2

I googled for a long time and tried to find something at https://www.sitemaps.org/protocol.html#index about gzip compression for sitemap INDEX files, but I couldn't find anything.

I have a very large dynamic site that is a directory of companies. I have 500+ sitemaps with 40,000 links each. All of these files are compressed with gzip and have the extension .xml.gz. A sitemap index file was created for these files, which I also decided to compress using gzip. Accordingly, its extension is also .xml.gz.

Now I am trying to index my site in the google search console. I add the sitemap index file to the sitemaps file section. The file is added, it is recognized as a sitemap index file, the status is Successful, but as a result, only 16 links from one sitemap0.xml.gz file were processed. At first I thought it was a temporary problem and they will be handled a little later. But I added this file back in April and I still have only 1 file out of 500+ processed and only 16 links from it.

At the same time, the console writes that it processed 400,000 links, and another 10 million links were found, but will not be indexed (without specifying any reasons, which I also do not understand).

So, looking for a bug, I want to ask if the sitemap index file can be gzipped or should it be left in its original xml form?

If possible, can someone tell me what to do? Just wait? I just haven't seen any changes in the console since April...

I honestly don't understand what the problem might be. Why were my 10 million links found but not indexed?

1
  • Current status from console: 425.000 links processed and indexed. 10.5 million links discovered - currently not indexed. Any help is appreciated Commented Aug 12, 2021 at 14:47

2 Answers 2

2

Sitemap index files may have .xml.gz extension with gzip compression. That isn't a problem.

Google Search Console has had a bug for years where it doesn't report all the URLs that it finds through sitemap index files. See Google says my sitemap index was "processed successfully", yet "total discovered URLs" is 0

There are many reasons why your pages might not get indexed. See: Why aren't search engines indexing my content? It is likely a combination of:

  • They are low quality -- It is nearly impossible to create 200 million high quality pages. They are either going to be sparse, have duplicate information from other sources, or be machine generated.
  • They are low interest -- They are about things that don't get many searches.
  • They have low Pagerank -- They don't have many links from anywhere.

XML sitemaps are not going to solve your problem. Listing pages in XML sitemaps does not usually get Google to index them. In the cases in which Google does index them, they won't rank better because they are in a sitemap. At best XML sitemaps get Googlebot to crawl your pages and give you extra stats in Google Search Console. See The Sitemap Paradox

5
  • Thank you for your answer. If my pages from 10 millions links is a catalog card information does it mean that the page is of low quality or of little interest? How then does Google suggest that I can increase interest in the directory pages? I have publications in well-known publications, but they all lead to the main page of my site, since this is directly a directory with a search on the main page of my site. How can I get search results for different companies from my directory if they are not indexed? Commented Aug 12, 2021 at 15:44
  • Getting deep links from other sites is good, but you also need to link the pages on your own site together. Commented Aug 12, 2021 at 15:49
  • But how can I link these pages if they are directory search results? how are the people in the phone book related? I can't cross-reference them even if I want to :( Commented Aug 12, 2021 at 15:55
  • There are all sorts of ways that businesses are related to each other: nearby, customer base, products sold, size, etc. Commented Aug 12, 2021 at 16:31
  • Okay, I understand you. Thank you very much again for your answer! Commented Aug 12, 2021 at 16:34
1

For those that arrive here from Google.

I've audited many large websites using the sitemap_index structure for indexation problems, and if pages in compressed sitemaps are not being indexed the first thing I check is the headers.

The problem is almost always that the Content-Type header is being served as text/xml, despite the fact that the content is compressed.

The Content-Type header for https://example.com/sitemap_index.gz should be application/gzip.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.