2

HI there over the last few days Google Webmaster Tools has been showing me 403 errors for a series of URLS. However when i FETCH them they come up as 200 OK and when I look in Google using keywords they appear in the index. Everything seems fine except for the errors. Is there any reason for this? Is there anywhere I should check for the cause of this. It is a wordpress site.

4
  • What does WMT mean? Your URLs are responding properly. $ curl -I http://www.mutantspace.com/daniel-palacios-waves-art-installation/ HTTP/1.1 200 OK. You can't control every 403 error. Some users have a bad network connection.
    – Chloe
    Commented Feb 27, 2015 at 18:41
  • WMT: google webmastwer tools. Thing is ive never got these reports before and i thought it meant that google cant crawl these URLs even though they turn up as 200 and are indexed. Some people have uggested im accidently firewalling bad connection from the other endgooglebot IPs but i dont know how to find out how...sghould i not worry? Are you saying that these 403s are a result of other users bad connection? Im a little confused. Thanks Commented Feb 27, 2015 at 18:59
  • @MorayBresnihan 403 status code errors indicate that the web server is denying a connection to the client, in this case the Googlebot. I'd suggest looking at your server's configuration files to make sure you're not inadvertently blocking certain IP's, domains, or user agents. Ask your web host provider for help if you don't know where to look for these.
    – dan
    Commented Feb 28, 2015 at 0:21
  • @MorayBresnihan Based on your comments below, it seems this was related to the IP address of Googlebot being blocked as suggested above. Let us know if you consider this resolved (so users don't continue to speculate further).
    – dan
    Commented Mar 1, 2015 at 6:15

4 Answers 4

3

Timing can be an issue.

In the time that you tested the URL's that turned up with error 403, they referenced files on the server with bad permissions. Make sure that the world has at least read and execute permissions to the file. In linux, you can use

chmod 755 (path/to/file with issue)

The fetch time must have happened AFTER setting the correct permissions.

Google also has a span of IP addresses reserved to itself for various operations and chances are you have a firewall installed that is blocking some of google webmaster tool's IP addresses.

If that doesn't work, look at all your apache logs and see if they report errors for certain file accesses. If there is anything unusual in your logs, then chances are your server is being hacked.

2
  • 1
    thanks for that Mike Ill try it out - i have a feeling its a firewall issue...ill also check permissions Commented Feb 28, 2015 at 9:43
  • Hi @Mike I just had a look at my error log and there are only 2 there. They state the following. Just wondering if this is the problem. If it is i have no idea how to solve it: [client 66.249.93.188] (12)Cannot allocate memory: couldn't create child process: /opt/suphp/sbin/suphp for /home/mutantsp/public_html/index.php, referer: mutantspace.com/konstantin-dimopoulos-blue-trees-installation Commented Feb 28, 2015 at 10:10
1

A 403 error is Forbidden. It can be for a series of reasons from file/directory permissions, not having DirectoryIndex in your configuration defined, or even a code error within your CMS/Blogging software. It has nothing to do with the network or your connection. In fact, the server responded didn't it? It simply means that the request you made was denied for a reason within the scope of a 403 error.

If you are finding intermittent errors, blame code first. What software are you using on your site? See if there is an update. I say this because I recently went through the same thing with the code on my site- and yes(!) I wrote it- Thank you very much. It was a code error that threw an exception within Apache that then results in a 403. While I or no-one can tell you why this is happening without putting our fingers on your keyboard, I would inspect my log file and see what requests were made and start from there. It took me a couple of days to figure out I was clobbering data between request processes when the requests per second count got rather high (in the multiple of dozens). The odds of clobbering data was infinitesimal even then and out of 68,000 requests I only got about 12 403's and only during very rapid request periods. Still- it was a code issue and simple to fix.

Otherwise, a 403 should be consistent and happen all the time.

4
  • thanks for that - ill go through logs and see what i can make of it Commented Feb 28, 2015 at 9:44
  • @MorayBresnihan 403s can be rather tricky to trace- unfortunately- though not always. In my case, it did take a me a couple of days. Let's hope your issue is a lot easier to solve. ;-) You may have to play detective and run a few tests to figure out under what conditions you get a 403. That's what I had to do. I knew a few things already, then I began answering questions with experiments to try and trace down the line to the actual problem.
    – closetnoc
    Commented Feb 28, 2015 at 17:26
  • well google webmaster tools has stopped telling me i have 403s. So thats good i suppose and ive deleted a few IPs out of my firewall. I did look at my error log and there was an issue with an IP that is WHOISed and found out was a googlebot. It was linked to a plugin. So i got rid of the incriminating timthumb code in the plugin that was causing it. So ill see how we go. Much appreciate the help Commented Feb 28, 2015 at 20:32
  • @MorayBresnihan For reference, I compiled a list of Google IP addresses from my database- it may help you to know what IP address ranges Google uses: webmasters.stackexchange.com/questions/71344/…
    – closetnoc
    Commented Feb 28, 2015 at 20:35
0

In response to your error log response, it seems like your problem is worse than what you're claiming it to be. based on whois records (go to whois.com), it turns out that the IP address you specified does belong to google. I pasted the first few lines of the result so you know what to look for:

    #
    # ARIN WHOIS data and services are subject to the Terms of Use
    # available at: https://www.arin.net/whois_tou.html
    #


    #
    # The following results may also be obtained via:
    # http://whois.arin.net/rest/nets;q=66.249.93.188?showDetails=true&showARIN=false&ext=netref2
    #

    NetRange:       66.249.64.0 - 66.249.95.255
    CIDR:           66.249.64.0/19
    OriginAS:       
    NetName:        GOOGLE
    NetHandle:      NET-66-249-64-0-1
    Parent:         NET-66-0-0-0-0
    NetType:        Direct Allocation
    RegDate:        2004-03-05
    Updated:        2012-02-24
    Ref:            http://whois.arin.net/rest/net/NET-66-249-64-0-1


    OrgName:        Google Inc.
    OrgId:          GOGL
    Address:        1600 Amphitheatre Parkway
    City:           Mountain View
    StateProv:      CA
    PostalCode:     94043
    Country:        US
    RegDate:        2000-03-30
    Updated:        2013-08-07
    Ref:            http://whois.arin.net/rest/org/GOGL

I'm actually surprised the server returned a 403 error. I thought in your error explanation, error code 500 would be returned.

Nevertheless, the REAL problem is that you're trying to run too many things on your server and you don't have enough memory to cover everything. You have a few options starting with the least recommended:

  1. Create a swap drive and set the size of it to at least the amount of extra memory you need and use it. I'd only recommend this if you are in an extreme hurry to have a website working in an OK state. The downfall to this method is that a disk is used for memory and accessing a disk is about 4 times slower than accessing memory.

  2. Cut back on running processes. If your server is meant to display web pages only and thats it, then make it so the mail server never starts along with other things you don't need.

If you use linux, then check the crontab and disable any programs there you don't need running by removing them. Also, use the PS command to see what processes are running and use the kill command to remove what you don't need running at one time.

If you are still low on memory, then check your web server. I'll assume apache. Check your httpd.conf settings and use a lower numbers for maxclients and serverlimit and then restart apache. (a graceful restart will not likely work here).

  1. Install more ram on the server.

Whatever you do, keep an eye every now and then on free memory and only load essential services. In linux, you can see free memory in megabytes with this command:

free -m

The 2nd item in the free column (on the line where it shows buffers/cache) is your free usable memory.

                total       used       free     shared    buffers     cached
    Mem:          1005        615        390          0         20        220
    -/+ buffers/cache:        373        631
    Swap:          243          0        243

On my system, I have 631 MB free usable memory out of 1005 MB. 390 MB hasn't been used by the OS at all.

Also, suphp is a handler for php which will execute one time per request, so you'll also need enough memory to run that as well.

2
  • thanks alot mike - very helpful. ill do exactly that although ill have to go learn about half of what you said....ill start with cron tab....and will go to my hosts and see what they can do....many thnaks again. I am getting very inconsistent 403s...much less the other day than earlier last week...so perhsps ill try turning off processes and getting rid of things i dont need... Commented Mar 2, 2015 at 19:06
  • went and had a look at the log resources on cpanel and noticed spikes in virtual memory to maximum on 24th and 25th February which is when this all started....its back down to normal - so would that cause it? Also wondering how so i do what you suggested 'If you use linux, then check the crontab and disable any programs there you don't need running by removing them. Also, use the PS command to see what processes are running and use the kill command to remove what you don't need running at one time.' thanks Commented Mar 2, 2015 at 19:31
0

Moray Bresnihan, checking crontab and using PS requires shell access to the server as the root user. This is achievable if you are renting a dedicated server (or equivalent). Then depending on your server setup, if it is secure, then you should have to login to the shell via SSH on a special port (other than 22) and use the credentials supplied by your dedicated server provider for any account other than root.

A good program for windows to use to get into your shell is putty. The download link is here:

http://www.chiark.greenend.org.uk/~sgtatham/putty/

Once logged in, use the SU command and type in the root password when prompted. then in there, you can use crontab -e to open the default editor with a list of items that will be executed at specified times.

If you happen to be unlucky and not have your own shell access, then you will need to contact your web space provider and ask them to do what I mentioned above for you.

Maxing out your memory to the peak is terrible regardless of how you look at it. You want to make sure you have some free bytes of RAM on your server when every process possible is running on your server. Failure to have free RAM on your server causes your server to run slower at best because swap space will then be used as memory and that in turn causes a disk to work harder than normal.

You may want to look into stress testing. This means accessing server email, accessing server ftp, and attempting to connect to a php page on apache at least a few hundred times a second (apache bench is a good tool for this). When you stress test, your memory usage will go up and down like wild because many processes start and stop on a frequent basis under a stress test.

3
  • thanks a million @Mike i have web provider working on it now hopefully they'll sort it out...ill stress test that all as well. And no i dont have a dedicated server - im kind of inbetween - a business shared host which gives me more space and some dedicated space....youve been a great help - i think i need to leaern a lot more about the mechanics of all of this as it takes os long to fix issues that are incomprehensible to me :) Commented Mar 3, 2015 at 14:40
  • Then if there are memory spikes and you're paying top dollar, then they can afford to add memory into their system can't they? If not that, then at least get a discount. Commented Mar 3, 2015 at 17:35
  • im still working on it - have deleted some plugins that were a bit heavy and talking to the host providers - in fairness to them their support is great but yep i might need to buy more memory... Commented Mar 4, 2015 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.