3

Google Search Console is telling me:

Access Denied

Googlebot couldn't crawl your URL because your server either requires authentication to access the page, or it is blocking Googlebot from accessing your site.

The URL looks something like this:

https://www.example.com/.html

I checked the URL in question and my server does indeed reply with a 403 Forbidden response and page.

This is not a URL in our sitemap.xml and we have never used it for anything. As far as I know, it has never been valid.

I tried the "Mark as Fixed" button in the Google Search Console, but the problem returned.

Any ideas how to properly handle without any detrimental effect to "link juice?" Thanks!

EDIT:

I believe I found the problem. This line in our Apache conf file:

  RewriteRule htm$ /$1.html [R=301,L]

There are no regex parentheses, so $1 will always be empty. So I believe the problem is caused by old backlinks hitting deprecated .htm URLs.

I replaced it with this:

  RewriteRule (.*)\.htm$ /$1.html [R=301,L]
  • 1
    This is likely from a bad link either on your site or another site. Do not marked as fixed. There is nothing to fix. This has nothing to do with link values that should worry about. Ignore it. – closetnoc Apr 27 '18 at 0:03
  • @closetnoc Thanks. Happy to ignore it but I guess it'll show as "Access denied" on the "URL Errors" report for all time? That doesn't seem fair to people with OCD. – Timothy B. Apr 27 '18 at 20:37
  • 1
    Lol! Call it therapy. I use sawmill for my log file analyzer and I just filter the junk. Wha la! I can be anal about some things too!! So I get it! – closetnoc Apr 27 '18 at 20:44
  • 1
    It is pretty much impossible to have a perfectly clean error report in Google Search Console. Google expects sites to respond with errors to nonsense URLs. A 404 Not Found error would be technically more correct than a 403, but it shouldn't make any real difference to Google. You'll just have squish your desire for order. I'm sorry, but life is unfair. – Stephen Ostermiller Apr 30 '18 at 15:14
3

Your server is probably configured to reject (with "403 Forbidden") all requests for dot files, ie. all files starting with a dot. These are usually config files, like .htaccess and .htpasswd, which should not be accessible publicly under any circumstance. These are naturally "hidden" on Linux, but not on Windows OS.

If you have access to the server config, then you can configure this, but otherwise this is normal behavior. Sometimes the server is configured to only block .ht* files, but otherwise a blanket "all dot files" should be blocked.

I tried the "Mark as Fixed" button in the Google Search Console, but the problem returned.

Unless you've changed the response on your server then this is not "fixed" and should not be marked as such in GSC.

This is not a "problem".

| improve this answer | |
  • 1
    I agree with this answer. Google expects all websites to have some errors. Having errors in Google Search Console in no way prevents the parts of your website that work from getting indexed and ranking well. Google alerts you to errors in case there is a problem you need to fix. If you never meant to have content on that URL, then it doesn't matter if there is a forbidden error there. For all Googlebot knows that is a password protected admin portion of your site. Having a 403 forbidden error on a junk URL will in no way hurt your site. – Stephen Ostermiller Apr 30 '18 at 15:09
0

This may be because of some wrong url rewrite condition.

To ignore it for the time being until you find the reson of this link.

You can do 301 (Moved Permanently) redirect for this particular url to home page.

| improve this answer | |
  • 1
    Redirecting URLs to the home page will cause Google Search Console to report them as "soft 404". The page would still be reported as an error. I don't ever recommend redirecting anything to the home page, it is better to let the URL give an actual 404 error. If they can't figure out why it is forbidden instead, that isn't so bad either. It won't hurt their site. – Stephen Ostermiller Apr 30 '18 at 15:11
  • @StephenOstermiller This used to be my SOP and I recently abandoned the practice. Thanks for all of your input. I don't have the ability to upvote comments or I would. – Timothy B. Apr 30 '18 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.