Recently I sent a sitemap to Google Search Console and I got some warnings saying that some urls are blocked by the robots.txt. I checked it and there is no single line that blocks the corresponding URLs.

However all the URLs shown in warning start with dash - character.

User-agent: *
Disallow: /Account/
Disallow: /random/
Disallow: /Random/
Disallow: /create/
Disallow: /login/
Disallow: /Login/
Disallow: /Edit/
Disallow: /edit/

This is my up-to-date robots.txt. Is there something about dash ("-") character related to robots.txt?

Screenshot of warning panel

Update: The issue is solved without us doing nothing. It turns out that even though Google Search Console shows everything okay, robots.txt is not updated internally for a while. It took 16 days to "really" update it.

  • 1
    "all the urls shown in warning start with dash" - all the URLS in your screenshot appear to start /watch/--? Has your robots.txt file changed in this time? – MrWhite May 9 '16 at 13:06
  • 1
    Are these valid URLs for your site?? Are they really in your sitemap? If so, check the source code of these pages for a reference that is blocked. It may not be the page per se' but a reference on the page. Otherwise, if these are not valid URLs, you may have another problem. – closetnoc May 9 '16 at 13:36
  • 1
    @w3dk You are right. What I was trying to say is that all the problematic urls are like "watch/--" or "watch/-" . If it is not a dash after the watch directory then everything is fine. – ozgur May 9 '16 at 14:41
  • 1
    @w3dk As a matter of fact. I tried robots.txt test tool of Search Console and it says "allowed" for those urls. They are allowed. But the warning remains the same although I resubmitted the sitemap couple of times. – ozgur May 9 '16 at 16:44
  • 1
    @w3dk Looking at the server history I can say it has changed 16 days ago. And there is in fact "/watch/" directory in it (don't ask why). But the point is, for the last 16 days robots.txt is as I stated in the queston. This is for sure. – ozgur May 10 '16 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.