1

Google's Webmaster tools shows lots of Crawl Errors. They are reported as 500 response code.

How can I go about diagnosing this issue?

The website is eCommerce business using the Magento platform.

Here is the notice from Google.

Google detected a significant increase in the number of URLs we could not access. It is likely that your server had an internal error or was busy when attempting to process these requests.

Recommended action

  • Check the Crawl Errors page in Webmaster Tools.
  • Check your scripts and script permissions.
  • Examine the log files on the server for your site for scripts or pages that might be crashing.
  • Consider addressing the load on your server.

Please see check Screenshot. enter image description here

Improve this question
2
  • There is not enough information to address this question. A 505 is not a temporary situation. You have a real and consistent issue that needs to be addressed. It can be a PHP application/permissions/installation issue or an invalid request that caused Apache to barf. 500 errors are not always a web application error. You will need to look in your error log file and look for the 500 errors to know more about what is failing. Post a few of then here and perhaps we can give you better advise. Also give us a clue as to your configuration, OS, web server, and what applications you have installed.
    – closetnoc
    Jun 27, 2014 at 14:35
  • If you visit the same page in a browser do you get a 500 error? If not, if you use WMT's Fetch as Googlebot do you get the error? Do you see 500 errors in your server's logs.
    – jeffatrackaid
    Jun 27, 2014 at 14:44

1 Answer 1

Reset to default
1

You should check the site to confirm if the errors are happening or not. Also look at the server logs to see if something happened at the time Google was crawling the site and then resolved itself. It could be an intermitted error that resolved itself but might show up again.

So steps to take:

  • Confirm if errors exist on the site for the pages reported in Webmaster Tools
  • Check the server logs for any errors
  • Check the database logs to see if the errors were at the database level
  • Review server load to see if there may have been a high load at the time and that caused the the issues
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.