Duplicate content issue after URL-change with 301-redirects

Question

We got the following problem:

• We changed all URLs on our page from oldURL.html to newURL.html and set up 301-redirects (ca. 600 URLs)
• Google re-crawled our page, indexed all the new URLs (newURL.html), but didn't crawl the old URLs (oldURL.html) again, as there were no internal links pointing at those domains anymore after the URL-change.
• This resulted in massive ranking-drops, etc. because (i) Google thought oldURL.html has exactly the same content as newURL, causing duplicate content issues, and (ii) Google did not transfer the juice from oldURL to newURL, because the 301-redirect was never noticed.
• Now we reset all internal Links to the old URLs again, which then redirect to the newURLs, in the hope that Google would re-crawl the pages, once there are internal links pointing at them.
• This is partially happening, but at a really low speed, so it would take multiple months to notice all-redirects. I guess, because Google thinks: "Aah, I already know oldURL.html, so no need to re-crawl it.

Possible solutions we thought of are ...

• Submitting as many of the old URLs to the index as possible via Webmaster Tools, to manually trigger a crawl. Doing that already
• Submitting a sitemap with all old URLs - but not sure if good idea, because Google does not seem to like 301-redirects in a sitemap
• ...

Both solutions are not perfect - and we cannot wait for three months, just to regain our old rankings. What are your ideas?

Best, David

Answer 1

You're proceeding from a misunderstanding:

"This resulted in massive ranking-drops, etc. because (i) Google thought oldURL.html has exactly the same content as newURL, causing duplicate content issues, and (ii) Google did not transfer the juice from oldURL to newURL, because the 301-redirect was never noticed."

• If 301 redirects are in place, on a 1:1 page-by-page level, Google doesn't think there's a duplicate content problem, because your server is telling Google that A has been superseded and replaced permanently by B.

• The 301 will be noticed, and "juice" will transfer. Just not overnight, and possibly not by a factor of 100%.

What you'd done sounded fine. Google will visit your site and index your new URLs, the redirects ensure that bots following external links to your site end up in the right place, etc. You could perhaps further expedite the process by submitting your old Sitemap alongside a new Sitemap, both referred from a Sitemap index, to be 100% sure Google hits all of your old URLs and follows their respective redirects (but as discussed in comments below, this shouldn't be necessary - rule out other problems first).

The fact that you've rolled back won't have helped matters any I'm afraid, but shouldn't be an insurmountable problem. Just be thorough with your redirections, and be patient while things settle down afterwards.