Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm facing the same problem with Googlebot not able to crawl my page. Mine problem doesn't have anything to do with www or not. Google can crawl all my pages except one directory and this directory contains all images, no code. (I have other images directory and Googlebot can crawl fine).

When checking Google Webmaster Tools for Crawl Error, the directory which Googlebot couldn't crawl and yield "Access Denied" error is: http://www.artistviewpoint.com/picture-to-painting/pictures/samples/.

Reading the guide from Google, I have already checked my robots.txt file and there is no log-in authentication of any sort. The only remaining tips which Google provide is to check the hosting provider or authentication using proxy which I do not know how to check.

share|improve this question
    
When I access that page from my home Internet I got "access denied" error too. Can you access it yourself!? –  Pazis Nov 27 '13 at 9:14
    
Is there any mention of the URL in htaccess (if you're using Apache)? –  Ivo van der Veeken Nov 27 '13 at 10:29
add comment

1 Answer 1

Its not just Googlebot that can't visit that page. If I visit in my browser I get a blank screen. If I try accessing it with curl on the command line I get:

$ curl --head http://www.artistviewpoint.com/picture-to-painting/pictures/samples/
HTTP/1.1 403 Forbidden
Content-Length: 0
Server: Microsoft-IIS/8.0
X-Powered-By: ASP.NET
Date: Sat, 30 Nov 2013 11:28:25 GMT

Google is not saying that they can't access any of the images in this directory. They are saying that they cannot view an index page for this directory. This is not really a problem. Configuring your server to deny access to a list of all your images is a very common setup. The error isn't going to prevent Googlebot from accessing, indexing, or ranking the rest of your site.

If you really want to get rid of the error you could:

  • put an index.html file in that directory
  • configure your server to redirect a request for an index.html file in that directory to your home page.
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.