Am I correct in thinking that having 23% of all crawl requests being blocked by "Unauthorized" 401/407 is NOT good?

68% of my crawl requests return 200 (OK) which is good of course.

Is this an urgent thing to fix? I'm thinking that perhaps those directories ought to be blocked by robots.txt?

  • Why would it be "bad"? Presumably you are intentionally blocking this content? Why are you blocking it? robots.txt simply prevents (good) bots from crawling content, not users - which is not the same thing. (But blocking with robots.txt vs 401 is not going to make any difference SEO-wise.) – MrWhite Apr 19 at 8:24
  • Why is this tagged "google-anyltics"? Most bots don't run JavaScript and their hits are never measured by GA. You also wouldn't normally include the GA tracking snippet on 401/407 Unauthorized pages. Are you actually trying to use GA to measure this? – Stephen Ostermiller Apr 19 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.