0

Recently we received a notice from Google Webmaster Tools stating "Googlebot found an extremely high number of URLs on your site:..."

We noticed that several of the sample urls listed included urls to pages with 400 HTTP status codes for malformed searches / requests on our site. What would be an appropriate solution to prevent Google from picking up on these pages? Should we change the HTTP status to a 404 instead? We are also considering using a meta content="noindex" tag.

Thanks in advance for your help!

  • 1
    I would rather think solving the problem first would be in order. Why is Google alerting you to the high number of URLs on your site getting a 400? What is the issue there? If there was a restructuring of the site and there should be large number of pages not found then a 404 or 410 would be appropriate. Otherwise, this sounds like a serious problem to be solved first and 400 vs. 404 is not even the issue. – closetnoc Sep 22 '15 at 1:20
1

First, you need to assess the URL's that google complains about.

If the URL's contain a bunch of gibberish characters and doesn't seem to fit in with your site, then I'd leave them as 400 status and remove the affected URLs from any sitemaps you have submitted to google. Examples of qualifying URLs should include:

http://example.com/%20%34%FA%C4aaa%FF/%F3
http://example.com/<>/</>/?/skfhsfh

If the URLs contain normal characters and they blend in with your site but the pages at those URLs no longer exist, then the 410 status should be returned so you won't waste google's time trying to index useless pages.

If the URLs are meant to point to existing pages at some point in the future but not at this time, then the status returned should be 404.

Examples of URLs with normal characters are:

http://example.com/mypage/page-1
http://example.com/fruits/bananas
http://example.com/fruits/apples.html

I recommend using noindex on thin-content pages returning status 200 (pages with little content) you don't want search engines to index at all.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.