I've built a small site for a local trader which is not a complicated site in any way. In fact it is 100% static. I have a few years experience and in that time have never encountered an error with Google Search Console where my
sitemap.xml have always been accepted and the pages properly indexed.
On this most recent site, the
sitemap.xml are both fully verified and accepted but I am receiving redirect errors on some of the most important pages and I don't understand why. I have not set up any intentional redirects. Importantly, the main index page has been indexed at Google with no problems, the difficulty only seems to be with pages that are inside folders.
The error (appears when inspecting url's in Google Search Console and it reads as follows)
- Headline URL is not on Google : Redirect Errors It cannot appear in Google search results until it's indexed. - Further details Crawl Allowed Page Fetch : Redirect Error
My guess work
Maybe on this occassion I have entirely screwed up the robots and sitemap but I do not know for sure so here they are for you to check.
I do also suspect that the SSL security certificate may have something to do with this issue but I am struggling to make sense of it as by all accounts the following files are acceptable with zero errors or warnings. There has never been a http version of the site, the errors are appearing even though it's a new upload with https installed prior to dealing with Google.
Please note that in the actual files a real URL is used. I have ommitted it from this post though so as not to spam inadvertently.
Robots.txt (somebody has pointed out that the URLS should not be detailed this way, this is just the latest thing I have tried, more details below).
Sitemap: https://-----.co.uk/sitemap.xml User-agent: * Allow: https://-----.co.uk/fitting/ Allow: https://-----.co.uk/carpentry/ Allow: https://-----.co.uk/utilities/ Allow: https://-----.co.uk/contact/ Disallow: https://-----.co.uk/assets/ Disallow: https://-----.co.uk/charity/ Disallow: https://-----.co.uk/coronavirus/ Disallow: https://-----.co.uk/privacy/ Disallow: https://-----.co.uk/dev/ Disallow: https://-----.co.uk/contact/PHPMailer-Master/ Disallow: https://-----.co.uk/contact/contact.php/
sitemap.xml (edited to exclude disallowed folders)
<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>https://-----.co.uk/</loc> <lastmod>2022-03-25T12:39:57+00:00</lastmod> </url> <url> <loc>https://-----.co.uk/carpentry</loc> <lastmod>2022-03-25T12:39:57+00:00</lastmod> </url> <url> <loc>https://-----.co.uk/contact</loc> <lastmod>2022-03-25T12:39:57+00:00</lastmod> </url> <url> <loc>https://-----.co.uk/fitting</loc> <lastmod>2022-03-25T12:39:57+00:00</lastmod> </url> <url> <loc>https://-----.co.uk/utilities</loc> <lastmod>2022-03-25T12:39:57+00:00</lastmod> </url> </urlset>
So, initially, I looked around Stack Overflow. I found some posts but none seemed relevant.
To keep trying for myself....
Every folder on the site was detailed in the sitemap.xml and the robots.txt but naturally in the latter some folders are detailed as
Disallowed. Initially I thought that was how the problem was arising so I removed the disallowed folders from the sitemap leaving only the allowed folders. This has made no difference.
I have also tried detailing pages themselves in both files as opposed to just a folder location. EG, I have tried
----.co.uk/carpentry and I have tried
On the topic of being canonical, As you can see full URL's have been used but I have also tried the alternative. EG just detailing the location of the sitemap and using
With regard to my expected result. Ideally I need Google to actually see there are no redirects. I simply cannot understand why these two files are verfied and the main index page is properly indexed but Google can't find its way into any of my folders.
Completely stuck, no idea what's going wrong. Could really do with moving on.