Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.
Sign up to join this community
I was submitting my sitemap in Google Webmasters and got this message when I clicked on the "test sitemap" button:
-Warning - URL blocked by robots.txt.
When I located my robots.txt file, this is what I saw in the robots.txt file:
# BEGIN GENESIS SEO
User-Agent: *
Disallow: /category
Disallow: /tag
Disallow: /author
Disallow: /?s=*
# END GENESIS SEO
Does anyone know what is the implication of this and how I should fix it please?
Your robots.txt tells search engines which parts of the site to ignore (and that the other parts are fair game). Your sitemap.xml tells search engines what you consider to be some of your most important parts of the site, and that you want those in the index.
When your sitemap includes URL's that your robots file disallows, you get that warning, because you asked to index something you disallowed.
Regardless of what's in your sitemap, Googlebot and Bingbot won't index it, if it's blocked in the robots file.
To achieve parity between the two, either remove the blocked URL's from your sitemap before re-submitting, or remove the block on those URL's from your robots file. Which path you take is up to you and depends on your website and its objectives.
Blocking tag, categories etc pages with a robots.txt file is a bad practice. Because most likely these pages are accessible on the website as links. Use Meta Robots tags instead for all these pages and exclude from sitemap.xml
To avoid Google Warning for your sitemap and get some "respect" from Google to your sitemap.xml file, at least remove your tag, categories etc. pages from the sitemap.xml. In sitemap.xml can't be robots.txt blocked pages.
The implication is that links that are accessible by users (and thus bots) are being blocked.
To discover which robot.txt entries are causing this, try one entry at a time (as multiple entries may be causing this issue) and test your sitemap again.
Best guess is that either the /category or /tags links are being unnecessarily blocked.
robots.txtfile, it looks like you could perhaps just ignore this warning, or remove the entries from your "sitemap"? – DocRoot May 20 '18 at 17:55robots.txtentries you have shared prevent the same pages being indexed by multiple URL's which can be (didn't say is) bad for SEO if you do not have proper canonical references for all content. – Willtech May 21 '18 at 12:34