Sign up ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Here is the problem:

Let say I have The has an iframe with inside but the robots.txt contains:

Disallow: /article/ 

Will Google crawl inside the iframe at or it will block, because robots.txt disallow the folder /article?

share|improve this question

1 Answer 1

up vote 1 down vote accepted

Disallow: /article/ will block the URL (or page) article/ and all subfolders or subpages.

That's why, if the robots.txt of applies this directive for the user agent is googlebot, don't worry.

Here are Google instructions on this.

share|improve this answer
So does the iframe in /article/mydomain pointed on will also block? – Rei Mar 25 '14 at 10:02
/article/mydomain is considered as a subpage; therefore, it's blocked by robots.txt. – Zistoloen Mar 25 '14 at 10:04
ok does the google provide a document regarding that? – Rei Mar 25 '14 at 10:14
I have edited my answer to give you the link to Google support in this subject. – Zistoloen Mar 25 '14 at 10:27
Nice, Thanks @Zistoloen – Rei Mar 25 '14 at 10:28

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.