Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We have a weird SEO issue;

In Google Webmaster Tools we are seeing a huge spike in Not Found Server Errors.

There are a set of several hundred dynamic pages which we need to maintain (though not have indexed) that look like this;

www.site.com/en-GB/Pages/job-details.aspx?job-id=123 

But weirdly Google is generating thousands of 404 crawl errors trying to index URLs with this structure;

www.site.com/en-GB/Pages/job-id?/en-GB/Pages/job-details.aspx?job-id=172649 

There are no;

  • Internal links pointing to this or similar
  • External links that we can find pointing to this or similar
  • Errors in the sitemap.xml

In a possibly related development; the URL Parameters section of Webmaster Tools, it’s listing some really weird parameters like; /en-GB/pages/job-details.aspx Which clearly is actually a chunk of the URL.

We have tried excluding this and the ‘job-id’ parameter from Google by choosing Edit > Yes > No URLs Thought to no avail!

My questions are:

  1. Why would Google add a parameter mid-way through the URL and how do we stop it?
  2. Why would Google think that a chunk of the URL was a parameter and how do we stop it?

I’m pretty stuck on this, so your help is much appreciated!

share|improve this question
add comment

1 Answer

Answering this as maybe it will help someone in future.

Basically we appear to have inadvertently added code that renders different markup conditionally on the user-agent (or 'cloaking' to it's friends). There are add-ons for Chrome and Firefox that allow you test this

The version of the page rendered for;

Googlebot/2.1 (+http://www.google.com/bot.html)

Contains the offending links.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.