What to do with malformed URLs crawled by Google and other bots?

Question

Sometime back due to a bug on our website, we end up exposing below URLs to Google. By the time realised and fixed this issue on production. They were crawled by Google.

/10695-Dilemma%20in%20purchasing%20a%20new%20car.-p2.html

actual url

/10695-dilemma-in-purchasing-a-new-car-p2.html

After a fix, these URLs doesn't exist on the site anymore. But Google keeps crawling them which leads to 404 errors.

What to do in this case?

1. Should I handle these URLs and start redirecting to actual URL?
2. Can we ignore these errors? If we ignore, any potential downside or any impact on SEO?
3. Any way to tell Google not to crawl these URLs?

Answer 1

You should be able to say in Webmaster tool that you have “Fixed” the issue.

Before you do that, make sure the %20 URLs do not appear in your website, that there is NO internal link within your website you still links to that URL. Google does not just crawl URLs in your sitemap and uploaded to Webmaster tools but crawls your site instead.

I had a similar issue with a Ruby on Rails website, and we created dynamic links, and for some reason, we did not replace “ “ with “-“ in all the links displayed on the portfolio landing page. So the error (%20 URLs) kept popping up in Webmaster tools.

When we fixed that - and told Google about that, these URLs disappeared.

No forwarding required from Google Perspective. However, other people may be used the URL to link to your site - if this is the case, implement Permanent Redirects (301s) to the actual URL.

If you need additional help - let me know your web server, CMS or Application Framework :)

Answer 2

You can submit a request to Google to remove them.

• Removals menu on the left hand side.
• This only removes them temporarily
• If they receive a 404 after that, it becomes permanent.

To remove them permanently use the Removal Tool

Remove Outdated Content

Remove outdated content from Google Search Guidelines This tool works only for pages or images that have already been modified or removed from the web. To remove personal information or content with legal issues that still exist on a page, submit a legal request instead.

Note however that if there is still an internal link to that page or an external reference leads to a valid page, it may get indexed again.

Try a redirect rule

If all such urls have spaces in them, then you could use a redirect rule to point them to 404.