Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a domain that has "fallen off a cliff" for searches in Google. Searches that used to be in position 1-4 are now gone from page 1. The same search in Bing shows the typical position expected (top 5 results).

In reviewing Google Webmaster Tools, I am seeing two problems: 1. The Sitemap is reporting two errors: General HTTP error: HTTP 403 error (Forbidden) URLs not accessible

However, the URL they provide as "no accessible" is accessible. I can click the link Google provides and it works fine.

  1. There are 6,000 crawl errors of type 403. Again, most of these pages that have 403 are accessible in my browser (tried various browsers as well). About half are from January, the other half from November.

  2. There are no IP-specific firewall rules on ports 80 and 443 that could block the goolgebot

  3. Using the user agent switcher add-on for FF I confirmed that the page loads when the user agent is the googlebot
  4. I an confirm that most of the pages reported as 403 are accessible.
  5. A search of just "site:thedomain.com" does confirm there are over 9,000 in the index. But most searches don't return the site.

I believe the 403 issues are the cause of the fall in search rankings, but I can't seem to find any information online with ideas about how to address this.

Any ideas?

jpe

share|improve this question
    
Did you try entering one of the page that Google sees ad 403 in a server response headers check tool. seoconsultants.com/tools/headers See if it answers with a 403. –  Marco Demaio Feb 2 '11 at 16:08
    
If you are willing to share your website address, maybe we could take a look? Make sure you don't have some funky server code requiring specific headers (Accept-Language), cookies etc. –  Tom Feb 4 '11 at 0:34
add comment

1 Answer 1

up vote 3 down vote accepted

If you're already in Google Webmaster Tools console, try the 'fetch as Googlebot' function to see if that is also returning a 403.

Double-check your robots.txt file, then your page heads for incorrect robots meta, and your server headers for incorrect configuration.

If you find nothing wrong, it could be an issue with Googlebot at a time when there was some network issue. Re-submit the XML sitemap, ping, and wait/review.

share|improve this answer
    
This led to the issue. Thanks! –  Justin Feb 5 '11 at 0:46
add comment

protected by John Conde Aug 21 '13 at 20:03

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.