Google attempting crawl of URLs with parameter randomly placed in URL

Question

We have a weird SEO issue;

In Google Webmaster Tools we are seeing a huge spike in Not Found Server Errors.

There are a set of several hundred dynamic pages which we need to maintain (though not have indexed) that look like this;

www.site.com/en-GB/Pages/job-details.aspx?job-id=123 

But weirdly Google is generating thousands of 404 crawl errors trying to index URLs with this structure;

www.site.com/en-GB/Pages/job-id?/en-GB/Pages/job-details.aspx?job-id=172649 

There are no;

• Internal links pointing to this or similar
• External links that we can find pointing to this or similar
• Errors in the sitemap.xml

In a possibly related development; the URL Parameters section of Webmaster Tools, it’s listing some really weird parameters like; /en-GB/pages/job-details.aspx Which clearly is actually a chunk of the URL.

We have tried excluding this and the ‘job-id’ parameter from Google by choosing Edit > Yes > No URLs Thought to no avail!

My questions are:

1. Why would Google add a parameter mid-way through the URL and how do we stop it?
2. Why would Google think that a chunk of the URL was a parameter and how do we stop it?

I’m pretty stuck on this, so your help is much appreciated!

Answer 1

Answering this as maybe it will help someone in future.

Basically we appear to have inadvertently added code that renders different markup conditionally on the user-agent (or 'cloaking' to it's friends). There are add-ons for Chrome and Firefox that allow you test this

The version of the page rendered for;

Googlebot/2.1 (+http://www.google.com/bot.html)

Contains the offending links.