Unknown bot crawling the same broken (HTTP 400) URL over and over again. Different User Agent and different country of origin.

The Problem

It seems at least once a week we're getting a big burst of HTTP 400 errors being hit on our site (we have logging to inform us). We'll check the logs in the morning and there's anywhere between 50 - 200 hits onto this single URL /foo/bar/item/.

What We Know

This URL appears on almost every page of our site (product listings) but is always formed as /foo/bar/item/857398 with an integer item ID on the end. When it's hit without an ID it correctly throws a HTTP 400 Invalid Request.

It seems this is a spider of some sort:

  • It hits with different user agents, seeming to vary between IE6, Firefox 5 and opera 8
  • It hits in small bursts of 2 - 10 requests every 30 minutes
  • It doesn't run JavaScript, as I can't find any trace of it in Google Analytics
  • It doesn't request any images linked on the page, the logs just list page after page, with no image requests between
  • It's very often proxy-ed to lots of different countries (we use Geo IP to trace as far as possible from the header information)
  • It doesn't send any HTTP_REFERER headers to trace which page it picked the URL up from

We've placed this URL in robots.txt as /foo/ because none of that URL subset should be indexable (almost all of it requires login).

I'm lost after that, it's still hitting this same URL over and over, I'm guessing it's picking it up from each individual page and just trying to fetch it every time, there doesn't seem to be any intelligence in remembering which URLs don't work.

I know this is almost impossible to stop as it's a public facing website being accessed by anyone who cares, but does anyone have any suggestions?

I also can't understand what they're achieving with such an inefficient crawling algorithm, or could this be some other kind of bot?


Here the $_SERVER dump, with identifying information redacted, everything else is intact.

$_SERVER=array (
  'REDIRECT_SCRIPT_URL' => '/foo/bar/item/',
  'REDIRECT_STATUS' => '200',
  'AC_HEADERS' => '',
  'SCRIPT_URL' => '/foo/bar/item/',
  'HTTP_HOST' => 'www.example.com',
  'HTTP_USER_AGENT' => 'Mozilla/5.0 (Windows NT 5.1; U; en) Opera
  'HTTP_COOKIE' => 'frontend=sfasdfasdfasdfasdfasdfdsf;
  'HTTP_VIA' => '1.1 localhost',
  'HTTP_CONNECTION' => 'Keep-Alive',
  'PATH' => '/usr/local/bin:/usr/bin:/bin',
  'SERVER_SIGNATURE' => '<address>Apache/2.2.16 (Debian) Server at
www.example.com Port 80</address>
  'SERVER_SOFTWARE' => 'Apache/2.2.16 (Debian)',
  'SERVER_NAME' => 'www.example.com',
  'SERVER_ADDR' => '**.**.**.**',
  'SERVER_PORT' => '80',
  'REMOTE_ADDR' => '**.**.**.**',
  'DOCUMENT_ROOT' => '/var/www/example.com/website/',
  'SERVER_ADMIN' => '[email protected]',
  'SCRIPT_FILENAME' => '/var/www/example.com/website/index.php',
  'REMOTE_PORT' => '51735',
  'REDIRECT_URL' => '/foo/bar/item/',
  'QUERY_STRING' => '',
  'REQUEST_URI' => '/foo/bar/item/',
  'SCRIPT_NAME' => '/index.php',
  'PATH_INFO' => '/foo.bar/item/',
  'PHP_SELF' => '/index.php/foo/bar/item/'
  • Can we have some exemple of request with full useragent ?
    – j0k
    Commented Mar 5, 2013 at 9:07
  • Are you asking out of curiosity, as this does not appear to be a "problem"? The number of hits are relatively small and you are already correctly handling this with a 400 status and not serving any resources so I can't see that there is anything else that you need to do? You can try adding rel="nofollow" to the links (if not already), but I suspect that will be fruitless as well. You say that "almost all of it requires login", except presumably the /foo/bar/item/... URLs?
    – MrWhite
    Commented Mar 5, 2013 at 10:40
  • "Different User Agent and different country of origin." (w/the same behavior in each case) - sounds like your site has attracted undesirable attention. Consider employing some tactics for dealing with misbehaving robots.
    – danlefree
    Commented Mar 5, 2013 at 11:03
  • @w3d You presume correct, /foo/bar/item/324234 is publicly accessible, but everything else under /foo isn't. I agree it's not a serious problem, but something somewhere is obviously wrong, as this isn't desired behaviour from either side of the connection. I'll try the nofollow I'd not thought of that.
    – Paystey
    Commented Mar 5, 2013 at 13:42
  • @danlefree We're looking into exactly that now. Thanks.
    – Paystey
    Commented Mar 5, 2013 at 14:37

1 Answer 1


Ideally you'd set a 301 Redirect from /foo/bar/item/ to either your homepage or to the main list of products (like a category-type page). This means:

  1. Any robots will be automatically taken to a valid page
  2. Any users will be automatically taken to a usable page
  3. Your error log should be much cleaner
  4. Search engines will stop picking up broken pages

If you have Webmaster Tools set up then these probably show under Crawl Errors, so you could click the "Linked From" tab and see if any pages have linked to that URL directly and fix the links. Even after fixing any broken links, the 301 Redirect option is still worth keeping in place.

  • I appreciate the response, and forcing a 301 to a product listing might be something we do. But this isn't Google's bot so web master tools shows nothing about the errors, and Google correctly ignores it as it's in the robots.txt.
    – Paystey
    Commented Mar 5, 2013 at 13:40
  • Out of curiosity then, why are you returning 400 instead of 404? Then you could at least have a user-friendly message for anyone who does happen upon the page by mistake. Commented Mar 5, 2013 at 14:17
  • 1
    We use a 400 because the real URL should contain an ID on the end, and the URL without an ID doesn't actually appear anywhere on the site. It's a problem with the construction of the URL, not that content could not be found for it. It's semantically valid, but obviously not to this bot.
    – Paystey
    Commented Mar 5, 2013 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.