Googlebot received Access Denied when crawling a directory

Question

I'm facing the same problem with Googlebot not able to crawl my page. Mine problem doesn't have anything to do with www or not. Google can crawl all my pages except one directory and this directory contains all images, no code. (I have other images directory and Googlebot can crawl fine).

When checking Google Webmaster Tools for Crawl Error, the directory which Googlebot couldn't crawl and yield "Access Denied" error is: http://www.artistviewpoint.com/picture-to-painting/pictures/samples/.

Reading the guide from Google, I have already checked my robots.txt file and there is no log-in authentication of any sort. The only remaining tips which Google provide is to check the hosting provider or authentication using proxy which I do not know how to check.

Answer 1

Its not just Googlebot that can't visit that page. If I visit in my browser I get a blank screen. If I try accessing it with curl on the command line I get:

$ curl --head http://www.artistviewpoint.com/picture-to-painting/pictures/samples/
HTTP/1.1 403 Forbidden
Content-Length: 0
Server: Microsoft-IIS/8.0
X-Powered-By: ASP.NET
Date: Sat, 30 Nov 2013 11:28:25 GMT

Google is not saying that they can't access any of the images in this directory. They are saying that they cannot view an index page for this directory. This is not really a problem. Configuring your server to deny access to a list of all your images is a very common setup. The error isn't going to prevent Googlebot from accessing, indexing, or ranking the rest of your site.

If you really want to get rid of the error you could:

• put an index.html file in that directory
• configure your server to redirect a request for an index.html file in that directory to your home page.