Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have checked multiple times in page, and there are no recursive sort of links anywhere nor are there any links without base href but still I can see lot of links with 10+ level deep. like GET http://www.example.com/a/a/a/a/a/a/a/a/a/foo_bar_page which is invalid for me, but example.com/a/foo_bar_page is a valid page. I can temporarily block it in robots.txt but what if it starts requesting for b/b/b/b/ sort of links ? How can I stop this ?

share|improve this question
    
Care to share the site in question? and mind if I check? –  bybe Mar 26 '13 at 13:09
add comment

2 Answers 2

Sometimes Google does try to GET pages that simply don't exist, finding the cause can be troublesome. If these pages are returning a valid 404 response there is nothing to worry about. As I say in numerous answers on this site a site that returns a 404 response is not broken and Google has said many times that 404's are completely normally (within reason of course).

Obvious things to check is:

  • On page internal links

Less Obvious things to check:

  • Plugins
  • CSS
  • JavaScript

Less Obvious and Normally Ignored Situations

  • Off Page Backlinks (Sometimes people, or automated sites can back link to pages that do not exist - at some point Google may discover that link and crawl back to your site and do a GET on that page which doesn't exist, and again if a 404 is returned! nothing to worry about.
share|improve this answer
    
Thanks for your reply, could you please let me know if there is info on how Google creates these GET request(somewhere hidden in developers site perhaps ?) –  GoodSp33d Mar 26 '13 at 14:44
add comment

Check the Google Web Master Tools. If you are not registered with them, do so.

Once in Google Web Master Tools, expand the Heath section and click on Crawl Errors. You will then see a list of URL Errors on the right. Find the offending URL in the list and click on it. A popup will appear with information on the error. The most important one usually is Linked From. This will list who links to it and that is the page to check.

The page can be one of your own or an external one. If it is one of yours, obviously things should be easy to fix. If not, you have to figure out how to contact the site who is linking or block it if you are unsuccessful.

Given the URL in your example, I suspect you have a relative link where you should have an absolute one. Maybe a menu bug were the page is supposed to point to itself. Just a guess.

share|improve this answer
    
yes even I thought the same, checked for all links using Xenu Tool and all seemed to be having a proper base href or / –  GoodSp33d Mar 26 '13 at 14:46
    
@2-Stroker - That was only a guess. If you look at the GWMT as described, you will at least see WHERE the link is coming from. It may give you an answer or open up more questions :) –  Itai Mar 26 '13 at 14:50
    
yep that s the first place which I checked, it says it landed on a/a.../a page from z/z.../z page These pages simply dont exist, but somehow Google Followed it from z to a. Confused –  GoodSp33d Mar 26 '13 at 14:56
    
@2-Stroker - Sounds like you have some digging to do. You may have had the problem accumulate over time which means something probably started the recursion. I would check the discovery date on GWMT too because maybe you had an error which is no longer there but Google still remembers the link. –  Itai Mar 26 '13 at 15:18
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.