We have a weird SEO issue;

In Google Webmaster Tools we are seeing a huge spike in Not Found Server Errors.

There are a set of several hundred dynamic pages which we need to maintain (though not have indexed) that look like this;


But weirdly Google is generating thousands of 404 crawl errors trying to index URLs with this structure;


There are no;

  • Internal links pointing to this or similar
  • External links that we can find pointing to this or similar
  • Errors in the sitemap.xml

In a possibly related development; the URL Parameters section of Webmaster Tools, it’s listing some really weird parameters like; /en-GB/pages/job-details.aspx Which clearly is actually a chunk of the URL.

We have tried excluding this and the ‘job-id’ parameter from Google by choosing Edit > Yes > No URLs Thought to no avail!

My questions are:

  1. Why would Google add a parameter mid-way through the URL and how do we stop it?
  2. Why would Google think that a chunk of the URL was a parameter and how do we stop it?

I’m pretty stuck on this, so your help is much appreciated!

1 Answer 1


Answering this as maybe it will help someone in future.

Basically we appear to have inadvertently added code that renders different markup conditionally on the user-agent (or 'cloaking' to it's friends). There are add-ons for Chrome and Firefox that allow you test this

The version of the page rendered for;

Googlebot/2.1 (+http://www.google.com/bot.html)

Contains the offending links.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.