Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

All of a sudden I had about 819 errors pop up in my Webmaster Tools accounts, all of which are PHP files. Google has never indexed these before, because they aren't URLs. The main folder they are indexing happens to be the my /content/ folder which contains basically the relevant information of my site.

My question is: if I block these directories via robots.txt will it affect Google crawling my real pages?

share|improve this question
up vote 0 down vote accepted

No they won't, unless you have relevant pages under /content, you'll have no problem with Google crawling you current pages.

Add the entry Disallow: /content/ to robots.txt.

Google tries to crawl subdirectories from your URLs, i.e. if Google finds an internal or external link pointing to a page in your site at /content/foo/foo.html, it will try to crawl /content/foo and /content.

share|improve this answer
Thanks for the help Pascual, to clarify, i don't have pages in this directory, only the files which dynamically generate the pages. By blocking these files, will it then block the pages these files create? – Jimmy Sep 27 '13 at 13:57
Of course, when I mean pages I also mean php files, as they generate HTML and thus a page to Google. Blocking that subdirectory, you will block Google from indexing all the pages generated by the files under /content :D – I.G. Pascual Sep 27 '13 at 14:00
Ok, so adding this to robots.txt would be a terrible idea in my situation, being as my site is being generated from this subdirectory. I need to determine why these are all of a sudden being shown to the world, in order to address the root problem. Thanks! – Jimmy Sep 27 '13 at 14:10
No, I confused you, I explained myself wrong: Say your site is www.example.com, if your URLs look like www.example.com/content/..., you have a problem. But if your URLs don't look like www.example.com/content/..., but your php files are located in .../foo/bar/content/ directory then you are ok. Google cannot see your php files, just the URLs you generate with them. – I.G. Pascual Sep 27 '13 at 14:23
My URL's do not have >/content/< anywhere in them. >/content/< is a directory i see in my FTP, my URL's generally are >example.com/catalog/....< or >example.com/catalog/products/...< – Jimmy Sep 27 '13 at 14:32

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.