1

I was submitting my sitemap in Google Webmasters and got this message when I clicked on the "test sitemap" button:

-Warning - URL blocked by robots.txt.

When I located my robots.txt file, this is what I saw in the robots.txt file:

# BEGIN GENESIS SEO
User-Agent: *
Disallow: /category 
Disallow: /tag 
Disallow: /author 
Disallow: /?s=*

# END GENESIS SEO

Does anyone know what is the implication of this and how I should fix it please?

|improve this question
  • 2
    So, what is the question? Is it that you don't understand this warning? Or that you think GSC is in error and the "warning" is unfounded? Without knowing your site and the URLs in question, it's difficult to answer this question. From that robots.txt file, it looks like you could perhaps just ignore this warning, or remove the entries from your "sitemap"? – DocRoot May 20 '18 at 17:55
  • You might also be interested in 5 Reasons Why Genesis SEO Needs to Go Away – Andrew T. May 21 '18 at 4:32
  • It is quite likely that the robots.txt entries you have shared prevent the same pages being indexed by multiple URL's which can be (didn't say is) bad for SEO if you do not have proper canonical references for all content. – Willtech May 21 '18 at 12:34
3

Your robots.txt tells search engines which parts of the site to ignore (and that the other parts are fair game). Your sitemap.xml tells search engines what you consider to be some of your most important parts of the site, and that you want those in the index.

When your sitemap includes URL's that your robots file disallows, you get that warning, because you asked to index something you disallowed.

Regardless of what's in your sitemap, Googlebot and Bingbot won't index it, if it's blocked in the robots file.

To achieve parity between the two, either remove the blocked URL's from your sitemap before re-submitting, or remove the block on those URL's from your robots file. Which path you take is up to you and depends on your website and its objectives.

|improve this answer
1

Blocking tag, categories etc pages with a robots.txt file is a bad practice. Because most likely these pages are accessible on the website as links. Use Meta Robots tags instead for all these pages and exclude from sitemap.xml

To avoid Google Warning for your sitemap and get some "respect" from Google to your sitemap.xml file, at least remove your tag, categories etc. pages from the sitemap.xml. In sitemap.xml can't be robots.txt blocked pages.

|improve this answer
1

The implication is that links that are accessible by users (and thus bots) are being blocked.

To discover which robot.txt entries are causing this, try one entry at a time (as multiple entries may be causing this issue) and test your sitemap again.

Best guess is that either the /category or /tags links are being unnecessarily blocked.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.