1

Our site is built on WordPress.

Whilst in the development stage, we had the option to Discourage search engines from indexing this site checked (Settings Reading).

We have now made the site live and unchecked this option.

Yesterday I submitted a sitemap into Googles Search Console but a lot of the sitemap paths are coming up with the warning:

Sitemap contains URLs which are blocked by robots.txt.

As far as I can tell, none of the site (Apart from the /wp-admin/) URLs are being blocked by our robots.txt file. I have tested it in the Search Console and that is saying it is fine.

I read some articles which say that the file can be cached for a little while, but it has now been a day since submitting it.

Is there anything I am missing or can do to stop this warning being thrown?

Sitemap - https://www.justaccounts.com/sitemap_index.xml

Robots File:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

A few examples of the URLs which are being shown as blocked:

  • https://www.example.com/page-sitemap.xml
  • https://www.example.com/category-sitemap.xml
  • https://www.example.com/attachment-sitemap.xml
| improve this question | |
  • 1
    "A few examples of the URLs which are being shown as blocked" - but those are the URLs for the sitemap(s) themselves? "but it has now been a day since submitting it" - you may just need to give it more time. The robots.txt file can be cached for a day or so and the GSC reports are not real time. – MrWhite Jul 4 '17 at 8:43
  • 2
    Google will probably show that warning until next time it tries to crawl those URLs. It may take a month or more. – Stephen Ostermiller♦ Jul 4 '17 at 9:28
  • Try deleting your cache, including your entire website cache, which should also reset the cache on your Yoast plugin. Make sure your robots.txt appears as you intend it to. (Yoast sometimes has trouble with that, if you set your robots.txt through the plugin as opposed to via FTP.) Go to GSC and submit your index sitemap again manually. – Henry Visotski Jul 5 '17 at 17:44
1

In my case Google was refusing to use (without saying) the new robots.txt because of the Windows line endings. I used dos2unix command and submitted again.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.