Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We have a weird SEO issue;

In Google Webmaster Tools we are seeing a huge spike in Not Found Server Errors.

There are a set of several hundred dynamic pages which we need to maintain (though not have indexed) that look like this;


But weirdly Google is generating thousands of 404 crawl errors trying to index URLs with this structure;


There are no;

  • Internal links pointing to this or similar
  • External links that we can find pointing to this or similar
  • Errors in the sitemap.xml

In a possibly related development; the URL Parameters section of Webmaster Tools, it’s listing some really weird parameters like; /en-GB/pages/job-details.aspx Which clearly is actually a chunk of the URL.

We have tried excluding this and the ‘job-id’ parameter from Google by choosing Edit > Yes > No URLs Thought to no avail!

My questions are:

  1. Why would Google add a parameter mid-way through the URL and how do we stop it?
  2. Why would Google think that a chunk of the URL was a parameter and how do we stop it?

I’m pretty stuck on this, so your help is much appreciated!

share|improve this question

Answering this as maybe it will help someone in future.

Basically we appear to have inadvertently added code that renders different markup conditionally on the user-agent (or 'cloaking' to it's friends). There are add-ons for Chrome and Firefox that allow you test this

The version of the page rendered for;

Googlebot/2.1 (+http://www.google.com/bot.html)

Contains the offending links.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.