10

I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via myschool.edu/myusername/myfolder/myfile.html returns a 403 error.

And yet Google somehow managed to grab the contents of those private files and store them in its cache! How is this possible? [I've since removed those files, so I'm just curious how Google managed to do this.]

  • 2
    This belongs on Webmasters – RobertPitt Dec 4 '10 at 18:57
5

The most probable reason is that the pages won't return a 403 header.

You can check that using the Web Developer Toolbar in Firefox or Chrome. The tool is located under "Information" -> "View Response Headers".

Also, the way I create my error pages is:

  1. I create some dummy error page. Let's say 403.php.
  2. I create an actual error page. For example error403.php.
  3. On the dummy error page, I put the following code: <?php header("Location: /error403.php",TRUE,301); ?>
  4. In my .htaccess, I put the following:

    Options -Indexes

    ErrorDocument 403 /403.php

This adds all the redirects in a proper way and makes me sure I'm getting some juice from my error pages.

This can actually be extended in an extremely cool way if your website has a search engine which uses GET requests.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.