How to get Google to de-index all the old URLs that now return 404 status?

Question

We are working on a website which has changed all their URLs. Right now, whenever people search on Google, the old URLs appear in the search results are not working. The URLs show 404 errors.

The site has around 100,000 products so redirect all the old URLs is a long process.

Answer 1

You don't need to do anything to get those URLs removed from Google. Google will automatically remove the old URLs from its index the next time it crawls them. Your old URLs will start to get removed from Google's index within a day, but it may take as much as two months for Googlebot to crawl all 100,000 old URLs.

After crawling a 404 status URL, Googlebot gives it a 24 hour grace period and then removes it from the index. If you used "410 Gone" status for those URLs instead, Googlebot would remove them from the index as soon as they are crawled.

I'd advise you to work on getting redirects to the new URLs in as quickly as possible. Start with the old URLs that used to get the most search traffic and those that have external links pointing to them. Even if you were to redirect your 500 best pages right now, that would make a huge difference to your SEO and user experience.

If you leave those URLs as 404 status, Googlebot will continue to come back and check on them periodically forever. I have 15 year old "not found" URLs that occasionally get crawled by Googlebot. However, as long as the return 404 status, Google will never index them.

Answer 2

Even though if it's more then one lakh products redirect Old URL to New URL based on category or most relevant path you can choose any single method from below sample code for your website

#301 Redirects for .htaccess

#Redirect a single page:
Redirect 301 /pagename.php http://www.example.com/pagename.html

#Redirect an entire site:
Redirect 301 / http://www.example.com/

#Redirect an entire site to a sub folder
Redirect 301 / http://www.example.com/subfolder/

#Redirect a sub folder to another site
Redirect 301 /subfolder http://www.example.com/

#Redirect any file with the .html extension to use the same filename but use the .php extension instead.
RedirectMatch 301 (.*)\.html$ http://www.example.com$1.php

#You can also perform 301 redirects using rewriting via .htaccess.

#Redirect from old example to new example
RewriteEngine on
RewriteBase /
RewriteRule (.*) http://www.newexample.com/$1 [R=301,L]

#Redirect to www location
RewriteEngine on
RewriteBase /
rewritecond %{http_host} ^example.com [nc]
rewriterule ^(.*)$ http://www.example.com/$1 [r=301,nc]

#Redirect to www location with subdirectory
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_HOST} example.com [NC]
RewriteRule ^(.*)$ http://www.example.com/directory/index.html [R=301,NC]

#Redirect from old example to new example with full path and query string:
Options +FollowSymLinks
RewriteEngine On
RewriteRule ^(.*) http://www.newexample.com%{REQUEST_URI} [R=302,NC]

#Redirect from old example with subdirectory to new example w/o subdirectory including full path and query string:
Options +FollowSymLinks
RewriteEngine On
RewriteCond %{REQUEST_URI} ^/subdirname/(.*)$
RewriteRule ^(.*) http://www.katcode.com/%1 [R=302,NC]

Rewrite and redirect URLs with query parameters (files placed in root directory)

Original URL:

http://www.example.com/index.php?id=1
Desired destination URL:

http://www.example.com/path-to-new-location/
.htaccess syntax:

RewriteEngine on
RewriteCond %{QUERY_STRING} id=1
RewriteRule ^index\.php$ /path-to-new-location/? [L,R=301]
Redirect URLs with query parameters (files placed in subdirectory)

Original URL:

http://www.example.com/sub-dir/index.php?id=1
Desired destination URL:

http://www.example.com/path-to-new-location/
.htaccess syntax:

RewriteEngine on
RewriteCond %{QUERY_STRING} id=1
RewriteRule ^sub-dir/index\.php$ /path-to-new-location/? [L,R=301]
Redirect one clean URL to a new clean URL

Original URL:

http://www.example.com/old-page/
Desired destination URL:
http://www.example.com/new-page/
.htaccess syntax:

RewriteEngine On
RewriteRule ^old-page/?$ $1/new-page$2 [R=301,L]
Rewrite and redirect URLs with query parameter to directory based structure, retaining query string in URL root level

Original URL:

http://www.example.com/index.php?id=100
Desired destination URL:

http://www.example.com/100/
.htaccess syntax:

RewriteEngine On
RewriteRule ^([^/d]+)/?$ index.php?id=$1 [QSA]
Rewrite URLs with query parameter to directory based structure, retaining query string parameter in URL subdirectory

Original URL:
http://www.example.com/index.php?category=fish
Desired destination URL:
http://www.example.com/category/fish/
.htaccess syntax:

RewriteEngine On
RewriteRule ^/?category/([^/d]+)/?$ index.php?category=$1 [L,QSA]
example change – redirect all incoming request from old to new example (retain path)

RewriteEngine on
RewriteCond %{HTTP_HOST} ^example-old\.com$ [NC]
RewriteRule ^(.*)$ http://www.example-new.com/$1 [R=301,L]
If you do not want to pass the path in the request to the new example, change the last row to:

RewriteRule ^(.*)$ http://www.example-new.com/ [R=301,L]

#From blog.example.com -> www.example.com/blog/
retains path and query, and eliminates xtra blog path if example is blog.example.com/blog/
Options +FollowSymLinks
RewriteEngine On
RewriteCond %{REQUEST_URI}/ blog
RewriteRule ^(.*) http://www.example.com/%{REQUEST_URI} [R=302,NC]
RewriteRule ^(.*) http://www.example.com/blog/%{REQUEST_URI} [R=302,NC]

Redirect it accordingly as relevant 301 redirect is a more important part of re-designing process try to map all the old + new URL into the excel and redirect it accordingly it'll be easier and faster.

More info: https://moz.com/blog/make-or-break-your-site-migration

Answer 3

To explicitly ask google search engine to delete stale URL, make a request via this page google.com/webmasters/tools/removals

That's it.