How to remove indexed pages from google effectively

Question

My site was getting traffic and it was on the rise, then I added https and somewhere in that time I also created tons of extra link for google to crawl, thinking this will increase traffic in those area where there was no traffic.

It backfired, my traffic reduced from 900/day hits to some 300 now. I have implemented 301 redirect for subdomains (subdomain somehow had duplicate pages so removed it) and then there was PHP script where I created these extra links, I don't know what properly unindex them now.

It has been 5 days since I made those changes, still the indexed pages are still in 55k which is huge. My site originally had 11k index pages, then rose to 100k and now at 55k. How can I remove those 'extra links' from index that I had created using PHP Script. Note I had asked google to index the whole page that I created and index all the links coming out of it.

PHP Code

This is the code where I created these links (forcefully). Basically my website is city base. I am pulling all cities, and then force them to use all the tags I have for all the cities.

<?php
$result1 = getDistinctBizCities();

if($result1 != false)
{
  while ($row1 = mysql_fetch_array($result1))
    echo '<h2>Businesses in '.$city.'</h2>';
    $result = getTags();
  {
    $city = ucfirst($row1['city']);

    if($result != -1)
    {
      echo'<ol>';
      while ($row = mysql_fetch_array($result))
      {

        echo '<li><a href="/search.php?keywords='.$row['tags'].'&city='.$row1['city'].'">'.$row['tags'].' in '.$row1['city'].'</a></li>';     
      }
      echo '</ol>';
    }   
  }
}
?>

Answer 1

In my humble opinion and I have already tested it - the best way of removing pages from index comes in a few phases:

1. On the pages you want to remove from index implement no-index.
2. Create a temporary sitemap.xml with URLs of these pages.
3. Ping this sitemap.xml file to Google using GSC or http://www.google.com/ping?sitemap=URL/of/file

Answer 2

I would fire the 410 Gone as a header - so Google quickly learns these pages are away and removes them fast from index.

Like

<?php
     if ($_SERVER['REQUEST_URI'] == '(url-pattern-to-remove-1|url-pattern-to-remove-2)') {
       header ("HTTP/1.1 410 Gone");
     }
?>

Answer 3

To remove indexed pages from SERP first you need to delete those pages and set redirection to the contact us page or which ever particular page might be.

After the site gets crawled again check that particular page appears on search results or not.

If the page still appears on search results you need to request Google to delete the page with the help of GSC. Steps to be followed to delete the pages which are appeared on search results. 1. Login to your webmaster tools. 2. Go back to the previous version of webmaster tools (i.e old version) 3. On the left-hand side, a menu will appear. 4. Click on Google Index and select remove URL's. 5. Copy & Paste the URL and follow the instructions. 6. After a few days you can check the status whether the page is deleted or not.

Answer 4

You may remove URLs from Google temporarily by using the Remove URLs feature the the Webmaster tools. As of this date (Aug. 2019), this feature is still in the old Webmaster tools and can be found on the following URL:

https://www.google.com/webmasters/tools/url-removal?hl=en&siteUrl=https://example.com

^ Replace example.com with your domain. From the menu, it is location under Google Index > Remove URLs

If there is a pattern in your URL, you can remove the entire directory by selecting that option in the dropdown. For your example PHP code, this would be search.php

I should also note: This isn't neccessarily an index issue - the index is a symptom of the primary issue, which is your wasted crawl budget.

By adding your scripting, you have drastically increased the number of pages needed to crawl by the Googlebot beyond your website's crawl budget. This really makes a mess of things and makes it harder for your other content to be more sticky in SERPs as they have a lower priority in your site hierarchy now due to the flood of irrelevant pages.

If you were to view your crawl stats, located on https://www.google.com/webmasters/tools/crawl-stats?hl=en&siteUrl=https://example.com (again, old version of webmaster tools not yet present in the newer search console), you will almost certainly identify a sharp spike in crawl when you added the script, but the spike will nowhere be enough to crawl the sheer volume of pages you created programatically.

Please add a noindex directive on all pages contains search.php to ensure the URL removal process will go smoothly, so if Google recrawls those pages that they will not reappear in the index.