1

Solved: Pages were being blocked by meta robots deliberately

A lot of pages are being blocked in the robots.txt file and when I checked the file there were no indications of a rule that blocks these.

The robots.txt file is structured:

Sitemap: https://domain.com/sitemap.xml
User-agent: *
Disallow: /directory-1/
Disallow: /directory-2/
Disallow: /directory-3/
Disallow: /directory-4/
Disallow: /directory-5/

User-agent: Googlebot-Image
Disallow:

None of the directories in the robots file match the URLs that are being blocked.

I was wondering if the incomplete Disallow could be the issue, despite it being only applicable to Googlebot-Image?

2
  • 1
    How do you know something is being blocked? Nov 23, 2016 at 20:23
  • I found it in a Moz report but, I must have read it wrong as these pages are being blocked in the meta deliberately.
    – GrapeSoda
    Nov 24, 2016 at 14:01

1 Answer 1

1

I have just found that an incomplete Disallow allows all robots complete access.

Found the answer here: http://www.robotstxt.org/robotstxt.html

1
  • "all robots complete access" - well, in your case, it specifically allows the Googlebot-Image robot complete access, not "all robots".
    – MrWhite
    Nov 23, 2016 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.