Batch removal of URL from google index

Question

We accidentally created a number of links to the same page resulting in Google indexing what should be about 5000 pages as over 100 000.

We have subsequently fixed this up, returning 404's, but google is still testing these URLs - several hundred a day - and returning these as 404 errors in webmaster tools. I assume that if we waited google would eventually sort this out, but as this is not good for our google ranking - we effectively have 10's of URL for each page - I would like to look to accelerate this removal.

We are manually removing these URL's using Google Webmaster Tools, but this is very time consuming. Unfortunately the URL's are not easy to cover by a single directory as described here

Does anyone have suggestions on batch removal tools of URL?

Answer 1

As I see it, you basically have two problems:

1. Google keeps returning the mistakenly created duplicate URLs in search results, confusing users, and

2. Googlebot keeps trying to recrawl the duplicate URLs, slowing down the crawling of your actual pages.

The first problem is more serious, since you're actually losing visitors. The best way to solve it would be to return HTTP 301 permanent redirects from the accidentally created URLs to the correct ones, so that your visitors will be sent to the page they were looking for. This will also eventually result in search engines dropping the redirected URLs from their index.

(Using 301 instead of 302 redirects here is important, since search engines interpret a 301 redirect as an instruction to index the target URL instead of the source.)

For the second problem, I'd recommend creating (and regularly updating) an XML sitemap of your actual pages. This will not completely stop Googlebot from crawling the accidentally created URLs (and you don't want it to, since the bot needs to find the 301 redirects), but it will keep the bot informed about updates to your actual pages, so that it can recrawl them faster.

As long as your servers can keep up with the load, you may also want to temporarily increase the maximum crawl rate for your site in Webmaster Tools.

Answer 2

Use status code 410 as described here http://support.google.com/webmasters/bin/answer.py?hl=en&answer=1663419

If the page no longer exists, make sure that the server returns a 404 (Not Found) or 410 (Gone) HTTP status code. Non-HTML file (like PDFs) should be completely removed from your server.

Answer 3

You can remove individual links from within Webmaster Tools, but there is no way to do it for a lot of links at once.

If you know the address of the pages that were created by accident you could add them to your robots.txt file so that they wouldn't be indexed by Google.

Answer 4

We just had this problem..

It was really ugly, 60,000 pages of spam created because of loop hole in our spam filter. We manually deleted all the pages, causing 404 error to be present for Google.

Months later the ugly pages were still in Google SERPS.

We searched for bulk removal tools in G webmaster, no luck, clearly 1 by 1 removal was not suitable using Google's tool.

We decided the best way was to add all of the 404 pages to our robots.txt file to the disallow list, (read up on this if you're not sure what it is).

THE TRICK, is, to do this within minutes we did the following:

1. Go to Google Webmaster tools, crawl errors and there downloaded to csv the errors.

2. Open csv, highlight the column with the URLS

3. Paste the URLS into a plain text editor, (removes frames). You'll get a list of full URLS.

4. Now you need to change http :// www.yoursite.co.uk/page-you-want-to-delte into Disallow: /page-you-want-to-delete

5. So paste the list into MS WORD, or similar word processor

6. Go to 'edit' find replace, find, 'http :// www.yoursite.co.uk' replace with 'Disallow: '

7. Tweak until you've got it right, paste the results into a basic text editor to remove fancy text formatting.

8. Bang you have the URLS in a format ready to copy and paste directly to your robots.txt