0

All of a sudden I had about 819 errors pop up in my Webmaster Tools accounts, all of which are PHP files. Google has never indexed these before, because they aren't URLs. The main folder they are indexing happens to be the my /content/ folder which contains basically the relevant information of my site.

My question is: if I block these directories via robots.txt will it affect Google crawling my real pages?

Improve this question

1 Answer 1

Reset to default
0

No they won't, unless you have relevant pages under /content, you'll have no problem with Google crawling you current pages.

Add the entry Disallow: /content/ to robots.txt.

Google tries to crawl subdirectories from your URLs, i.e. if Google finds an internal or external link pointing to a page in your site at /content/foo/foo.html, it will try to crawl /content/foo and /content.

Improve this answer
6
  • Thanks for the help Pascual, to clarify, i don't have pages in this directory, only the files which dynamically generate the pages. By blocking these files, will it then block the pages these files create?
    – Jimmy
    Commented Sep 27, 2013 at 13:57
  • Of course, when I mean pages I also mean php files, as they generate HTML and thus a page to Google. Blocking that subdirectory, you will block Google from indexing all the pages generated by the files under /content :D
    – I.G. Pascual
    Commented Sep 27, 2013 at 14:00
  • Ok, so adding this to robots.txt would be a terrible idea in my situation, being as my site is being generated from this subdirectory. I need to determine why these are all of a sudden being shown to the world, in order to address the root problem. Thanks!
    – Jimmy
    Commented Sep 27, 2013 at 14:10
  • No, I confused you, I explained myself wrong: Say your site is www.example.com, if your URLs look like www.example.com/content/..., you have a problem. But if your URLs don't look like www.example.com/content/..., but your php files are located in .../foo/bar/content/ directory then you are ok. Google cannot see your php files, just the URLs you generate with them.
    – I.G. Pascual
    Commented Sep 27, 2013 at 14:23
  • 1
    My URL's do not have >/content/< anywhere in them. >/content/< is a directory i see in my FTP, my URL's generally are >example.com/catalog/....< or >example.com/catalog/products/...<
    – Jimmy
    Commented Sep 27, 2013 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.