Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We accidentally created a number of links to the same page resulting in Google indexing what should be about 5 000 pages as over 100 000.

We have subsequently fixed this up, returning 404's, but Google is still testing these URLs - several hundred a day - and returning these as 404 errors in Webmaster Tools. I assume that if we waited Google would eventually sort this out, but as this is not good for our Google ranking - we effectively have 10's of URL for each page - I would like to look to accelerate this removal.

We are manually removing these URL's using Google Webmaster Tools, but this is very time consuming. Unfortunately the URL's are not easy to cover by a single directory as described here

Does anyone have suggestions on batch removal tools of URL?

share|improve this question

4 Answers 4

As I see it, you basically have two problems:

  1. Google keeps returning the mistakenly created duplicate URLs in search results, confusing users, and

  2. Googlebot keeps trying to recrawl the duplicate URLs, slowing down the crawling of your actual pages.

The first problem is more serious, since you're actually losing visitors. The best way to solve it would be to return HTTP 301 permanent redirects from the accidentally created URLs to the correct ones, so that your visitors will be sent to the page they were looking for. This will also eventually result in search engines dropping the redirected URLs from their index.

(Using 301 instead of 302 redirects here is important, since search engines interpret a 301 redirect as an instruction to index the target URL instead of the source.)

For the second problem, I'd recommend creating (and regularly updating) an XML sitemap of your actual pages. This will not completely stop Googlebot from crawling the accidentally created URLs (and you don't want it to, since the bot needs to find the 301 redirects), but it will keep the bot informed about updates to your actual pages, so that it can recrawl them faster.

As long as your servers can keep up with the load, you may also want to temporarily increase the maximum crawl rate for your site in Webmaster Tools.

share|improve this answer

You can remove individual links from within Webmaster Tools, but there is no way to do it for a lot of links at once.

If you know the address of the pages that were created by accident you could add them to your robots.txt file so that they wouldn't be indexed by Google.

share|improve this answer

Use status code 410 as described here http://support.google.com/webmasters/bin/answer.py?hl=en&answer=1663419

If the page no longer exists, make sure that the server returns a 404 (Not found) or 410 (Gone) HTTP status code. Non-HTML file (like PDFs) should be completely removed from your server.

share|improve this answer
Thanks for the suggestion - we have done this (using 404's, but can easily change this to 410). The issue, though, is that regardless of 404's or 410's it will take a long time for Google to remove these pages. Google seems to do a few hundred pages a day - given that we have 100 000, this is several years. We want to accelerate this and request this removal, so hence my request for a batch removal tool. –  plantify Apr 21 '12 at 18:47
There is nothing in the Webmaster Tools API to remove pages I checked, that would have been your best option. The next best thing to do is create a PHP CURL script to login and remove them if it's cost effective. –  Anagio Apr 21 '12 at 19:09

We just had this problem.

It was really ugly, 60 000 pages of spam created because of loop hole in our spam filter. We manually deleted all the pages, causing 404 error to be present for Google.

Months later the ugly pages were still in Google SERPs.

We searched for bulk removal tools in Google Webmaster, no luck, clearly 1 by 1 removal was not suitable using Google's tool.

We decided the best way was to add all of the 404 pages to our robots.txt file to the disallow list, (read up on this if you're not sure what it is).

The trick, is, to do this within minutes we did the following:

  1. Go to Google Webmaster Tools, crawl errors and there downloaded to .csv the errors.

  2. Open .csv, highlight the column with the URLs

  3. Paste the URLs into a plain text editor, (removes frames). You'll get a list of full URLs.

  4. Now you need to change http://www.yoursite.co.uk/page-you-want-to-delte into Disallow: /page-you-want-to-delete.

  5. So paste the list into Ms Word, or similar word processor

  6. Go to 'edit' find replace, find, http://www.yoursite.co.uk replace with Disallow: .

  7. Tweak until you've got it right, paste the results into a basic text editor to remove fancy text formatting.

  8. Bang you have the URLs in a format ready to copy and paste directly to your robots.txt.

share|improve this answer
Just curious: why using WORD to find and replace? Any text editor can do that. –  MatteoSp May 4 '13 at 8:12

protected by John Conde Jun 18 '13 at 0:36

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.