Solved: Pages were being blocked by meta robots deliberately

A lot of pages are being blocked in the robots.txt file and when I checked the file there were no indications of a rule that blocks these.

The robots.txt file is structured:

Sitemap: https://domain.com/sitemap.xml
User-agent: *
Disallow: /directory-1/
Disallow: /directory-2/
Disallow: /directory-3/
Disallow: /directory-4/
Disallow: /directory-5/

User-agent: Googlebot-Image

None of the directories in the robots file match the URLs that are being blocked.

I was wondering if the incomplete Disallow could be the issue, despite it being only applicable to Googlebot-Image?

  • 1
    How do you know something is being blocked? – Stephen Ostermiller Nov 23 '16 at 20:23
  • I found it in a Moz report but, I must have read it wrong as these pages are being blocked in the meta deliberately. – GrapeSoda Nov 24 '16 at 14:01

I have just found that an incomplete Disallow allows all robots complete access.

Found the answer here: http://www.robotstxt.org/robotstxt.html

  • "all robots complete access" - well, in your case, it specifically allows the Googlebot-Image robot complete access, not "all robots". – MrWhite Nov 23 '16 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.