Here is the problem:

Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains:

Disallow: /article/ 

Will Google crawl mydomain.com inside the iframe at externaldomain.com/article/mydomain or it will block mydomain.com, because externaldomain.com robots.txt disallow the folder /article?

1 Answer 1


Disallow: /article/ will block the URL (or page) article/ and all subfolders or subpages.

That's why, if the robots.txt of externaldomain.com applies this directive for the user agent is googlebot, don't worry.

Here are Google instructions on this.

  • So does the iframe in /article/mydomain pointed on mydomain.com will also block?
    – Rei
    Mar 25, 2014 at 10:02
  • /article/mydomain is considered as a subpage; therefore, it's blocked by robots.txt.
    – Zistoloen
    Mar 25, 2014 at 10:04
  • ok does the google provide a document regarding that?
    – Rei
    Mar 25, 2014 at 10:14
  • I have edited my answer to give you the link to Google support in this subject.
    – Zistoloen
    Mar 25, 2014 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.