3

I've built a small site for a local trader which is not a complicated site in any way. In fact it is 100% static. I have a few years experience and in that time have never encountered an error with Google Search Console where my robots.txt and sitemap.xml have always been accepted and the pages properly indexed.

On this most recent site, the robots.txt and sitemap.xml are both fully verified and accepted but I am receiving redirect errors on some of the most important pages and I don't understand why. I have not set up any intentional redirects. Importantly, the main index page has been indexed at Google with no problems, the difficulty only seems to be with pages that are inside folders.

The error (appears when inspecting url's in Google Search Console and it reads as follows)

- Headline
URL is not on Google : Redirect Errors
It cannot appear in Google search results until it's indexed.

- Further details
Crawl Allowed
Page Fetch : Redirect Error

My guess work

Maybe on this occassion I have entirely screwed up the robots and sitemap but I do not know for sure so here they are for you to check.

I do also suspect that the SSL security certificate may have something to do with this issue but I am struggling to make sense of it as by all accounts the following files are acceptable with zero errors or warnings. There has never been a http version of the site, the errors are appearing even though it's a new upload with https installed prior to dealing with Google.

Please note that in the actual files a real URL is used. I have ommitted it from this post though so as not to spam inadvertently.

Robots.txt (somebody has pointed out that the URLS should not be detailed this way, this is just the latest thing I have tried, more details below).

Sitemap: https://-----.co.uk/sitemap.xml
User-agent: *
Allow: https://-----.co.uk/fitting/
Allow: https://-----.co.uk/carpentry/
Allow: https://-----.co.uk/utilities/
Allow: https://-----.co.uk/contact/
Disallow: https://-----.co.uk/assets/
Disallow: https://-----.co.uk/charity/
Disallow: https://-----.co.uk/coronavirus/
Disallow: https://-----.co.uk/privacy/
Disallow: https://-----.co.uk/dev/
Disallow: https://-----.co.uk/contact/PHPMailer-Master/
Disallow: https://-----.co.uk/contact/contact.php/

sitemap.xml (edited to exclude disallowed folders)

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
        <url>
        <loc>https://-----.co.uk/</loc>
        <lastmod>2022-03-25T12:39:57+00:00</lastmod>
    </url>
    <url>
        <loc>https://-----.co.uk/carpentry</loc>
        <lastmod>2022-03-25T12:39:57+00:00</lastmod>
    </url>
    <url>
        <loc>https://-----.co.uk/contact</loc>
        <lastmod>2022-03-25T12:39:57+00:00</lastmod>
    </url>
    <url>
        <loc>https://-----.co.uk/fitting</loc>
        <lastmod>2022-03-25T12:39:57+00:00</lastmod>
    </url>
    <url>
        <loc>https://-----.co.uk/utilities</loc>
        <lastmod>2022-03-25T12:39:57+00:00</lastmod>
    </url>
</urlset>

So, initially, I looked around Stack Overflow. I found some posts but none seemed relevant.

To keep trying for myself....

Every folder on the site was detailed in the sitemap.xml and the robots.txt but naturally in the latter some folders are detailed as Disallowed. Initially I thought that was how the problem was arising so I removed the disallowed folders from the sitemap leaving only the allowed folders. This has made no difference.

I have also tried detailing pages themselves in both files as opposed to just a folder location. EG, I have tried ----.co.uk/carpentry and I have tried -----.co.uk/carpentry/index.php.

On the topic of being canonical, As you can see full URL's have been used but I have also tried the alternative. EG just detailing the location of the sitemap and using /carpentry and /fitting etc.

With regard to my expected result. Ideally I need Google to actually see there are no redirects. I simply cannot understand why these two files are verfied and the main index page is properly indexed but Google can't find its way into any of my folders.

Completely stuck, no idea what's going wrong. Could really do with moving on.

Improve this question
4
  • 2
    You've added a good amount of details, and tried to avoid the pretext of spamming, but I think we're going to need the URL(s) that you're using in Google Search Console that's resulting in the redirect error. You can either take a screenshot of that in Google Search Console, or just add it here in the comments if you're comfortable with that.
    – dan ♦
    Commented Mar 30, 2022 at 16:03
  • 1
    Hiya Dan and thanks. I don't mind sharing the url. Genuinely did not want to spam. The website is at acmpropertyservices.co.uk which at the root dir seems to work but then inspecting all the folders detailed in my sitemap will produce the error.
    – Cutey from Cute Code
    Commented Mar 30, 2022 at 16:07
  • 1
    Your sitemap has /fitting in it, but when I visit the URL it redirects to /fitting/ (with a trailing slash). Which version are you inspecting in GSC, the URL with or without the trailing slash?
    – Stephen Ostermiller ♦
    Commented Mar 30, 2022 at 16:48
  • Hi again @StephenOstermiller Yes indeed. This has been pointed out in the answer below and I have now altered it and resubmitted the urls. I will be sure to update you once Google Console gives me some kind of result. Thanks for all your help today.
    – Cutey from Cute Code
    Commented Mar 30, 2022 at 16:49

1 Answer 1

Reset to default
4 
<loc>https://-----.co.uk/contact</loc>

By the look of your robots.txt file, it would seem /contact (and I assume the other "folders") are physical directories on the filesystem. In which case the canonical URL must end in a trailing slash (unless you take additional steps to workaround this). So, the URLs in your sitemap and all associated internal links need to end in a trailing slash.

If you don't include a trailing slash then mod_dir will issue a 301 external redirect to append the trailing slash (to "fix" the URL). This might be the erroneous "redirect" that Google is seeing.

Also make sure you are looking at (and submitting HTTPS sitemaps to) the HTTPS property in GSC, not the HTTP property (which is presumably redirected to HTTPS).

Allow: https://-----.co.uk/fitting/
Allow: https://-----.co.uk/carpentry/
Allow: https://-----.co.uk/utilities/
Allow: https://-----.co.uk/contact/
Disallow: https://-----.co.uk/assets/
Disallow: https://-----.co.uk/charity/
Disallow: https://-----.co.uk/coronavirus/
Disallow: https://-----.co.uk/privacy/
Disallow: https://-----.co.uk/dev/
Disallow: https://-----.co.uk/contact/PHPMailer-Master/
Disallow: https://-----.co.uk/contact/contact.php/

Yes, as you stated, the "the URLs should not be detailed this way". The Allow and Disallow directives take a root-relative URL, starting with a slash. So, in it's current state, your robots.txt file is not doing anything since none of the rules will match (so is not the cause of your immediate problem).

But your 4 Allow directives do not seem to be required here anyway. The default is to allow crawling and there is no conflict with the Disallow directives in what you have posted (which would warrant a need for a more specific Allow directive).

So your robots.txt file should look more like this:

User-agent: *
Disallow: /assets/
Disallow: /charity/
Disallow: /coronavirus/
Disallow: /privacy/
Disallow: /dev/
Disallow: /contact/PHPMailer-Master/
Disallow: /contact/contact.php/

Sitemap: https://example.co.uk/sitemap.xml

Note that the Sitemap directive does take an absolute URL.

Improve this answer
5
  • Many thanks Mr White. I have followed your recommends, altered my sitemap to include the trailing slashes and have resubmitted the pages. In relation to the allows and disallows. Initally I had only detailed the disallows. The current itterations of robots and sitemaps above pretty much just represent my latest efforts having not solved anything. I will now change that. I suspect Google may take a day or two now but I won't forget to come back and vote you up. Really hope this works. Thank you.
    – Cutey from Cute Code
    Commented Mar 30, 2022 at 16:18
  • @CuteCodeRob And make sure your internal links all reference the canonical URLs as well. I've updated my answer with an idea of what the robots.txt file should look like. There should be a line gap between rule-blocks.
    – MrWhite
    Commented Mar 30, 2022 at 16:25
  • Do you mean any internal links that are detailed in my HTML such as links to pages etc? I am really grateful for your assistance and I will be sure to get back to you as soon as my Search Console updates. Thank again.
    – Cutey from Cute Code
    Commented Mar 30, 2022 at 16:29
  • 1
    @CuteCodeRob Yes, all links in your HTML.
    – MrWhite
    Commented Mar 30, 2022 at 16:48
  • Thank you very much indeed, your suggestions have worked perfectly. I still have some errors but these are to do with the URL's that were wrongly submitted (no trailing slash). Fairly sure I don't need to as it's automatic but I have now submitted those URL's for removal.
    – Cutey from Cute Code
    Commented Mar 30, 2022 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.