I am, as a volunteer, taking care of our local YMCA websites (thereof refered to as [sub].domain where sub refers to different websites). For some reason, a few months ago google stopped indexing our websites. I have been trying to solve this ever since, but without any luck.
- I (sometimes) cannot verify ownership: the requests always times out. Interestingly enough, it sometimes work. And sometimes not. I have not been able to spot any pattern; I added one website successfully (subdomain), but not the other (main domain)
- Google seems not to be able to read
robots.txt- I have tried accessing it through the browser, command line tools to inspect redirect chains, and other validator services, and all seems fine
- Google validator says it cannot be read at the moment, and that it will use older version if acccessible; the tool shows that a crawler can access URLs I've tried out
- Requests for pages indexing fail because it says
robots.txtis not accessible
- Google seems not to be able to load
sitemap.xml- I have tried similarly as above all sorts of things, the search console shows 'general HTTP error'
- It should be already obvious that no pages are indexed - nothing works - at least wrt. google, all other validators, and services seem to work fine.
It can be this problem is also on other websites living on the same server, I cannot really say because of the first issue. I got the verification done only on the site I gave a link to.
It's as if google was blocked by our server. Websites are running on self-hosted Synology station, via Apache (that somehow operates through Nginx). Unfortunately, I do not have much knowledge nor direct access to these.
The mentioned file contents: robots.txt
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
sitemap.xml
<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="https://wp.[domain]/wordpress/wp-content/plugins/google-sitemap-plugin/sitemap-index.xsl"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>http://wp.[domain]/wordpress/sitemap_1.xml</loc>
<lastmod>2022-11-28T19:56:36+00:00</lastmod>
</sitemap>
<sitemap>
<loc>https://site1.[domain]/sitemap_2.xml</loc>
<lastmod>2022-11-28T20:05:53+00:00</lastmod>
</sitemap>
<sitemap>
<loc>https://site2.[domain]/sitemap_3.xml</loc>
<lastmod>2022-12-17T18:39:09+00:00</lastmod>
</sitemap>
</sitemapindex>
sitemap_3.xml
<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="https://site2.[domain]/wp-content/plugins/google-sitemap-plugin/sitemap.xsl"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>[url]</loc>
<lastmod>2022-12-17T19:39:08+00:00</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
...
</urlset>