1

e

My site was getting traffic and it was on the rise, then I added https and somewhere in that time I also created tons of extra link for google to crawl, thinking this will increase traffic in those area where there was no traffic.

It backfired, my traffic reduced from 900/day hits to some 300 now. I have implemented 301 redirect for subdomains (subdomain somehow had duplicate pages so removed it) and then there was PHP script where I created these extra links, I don't know what properly unindex them now.

It has been 5 days since I made those changes, still the indexed pages are still in 55k which is huge. My site originally had 11k index pages, then rose to 100k and now at 55k. How can I remove those 'extra links' from index that I had created using PHP Script. Note I had asked google to index the whole page that I created and index all the links coming out of it.

PHP Code

This is the code where I created these links (forcefully). Basically my website is city base. I am pulling all cities, and then force them to use all the tags I have for all the cities.

<?php
$result1 = getDistinctBizCities();

if($result1 != false)
{
  while ($row1 = mysql_fetch_array($result1))
    echo '<h2>Businesses in '.$city.'</h2>';
    $result = getTags();
  {
    $city = ucfirst($row1['city']);

    if($result != -1)
    {
      echo'<ol>';
      while ($row = mysql_fetch_array($result))
      {

        echo '<li><a href="/search.php?keywords='.$row['tags'].'&city='.$row1['city'].'">'.$row['tags'].' in '.$row1['city'].'</a></li>';     
      }
      echo '</ol>';
    }   
  }
}
?>

4 Answers 4

2

In my humble opinion and I have already tested it - the best way of removing pages from index comes in a few phases:

  1. On the pages you want to remove from index implement no-index.
  2. Create a temporary sitemap.xml with URLs of these pages.
  3. Ping this sitemap.xml file to Google using GSC or http://www.google.com/ping?sitemap=URL/of/file
4
  • This would require to make custom sitemap.xml as the ones available online creates a regular sitemap
    – TheTechGuy
    Mar 1, 2019 at 15:04
  • It would, but thats a 10 minute job... @slowdiver
    – user98424
    Mar 1, 2019 at 15:46
  • what about not xml page, regular links..will that work?
    – TheTechGuy
    Mar 1, 2019 at 16:04
  • 1
    You can use this for example: timestampgenerator.com/tools/xml-sitemap-from-list
    – user98424
    Mar 1, 2019 at 17:00
1

I would fire the 410 Gone as a header - so Google quickly learns these pages are away and removes them fast from index.

Like

<?php
     if ($_SERVER['REQUEST_URI'] == '(url-pattern-to-remove-1|url-pattern-to-remove-2)') {
       header ("HTTP/1.1 410 Gone");
     }
?>
7
  • This is the link to that page, what is your recommendation. Just remove the links and set the headers?
    – TheTechGuy
    Mar 1, 2019 at 9:48
  • Because these links are valid otherwise, but I forcefully created them, do not have any valid contents.
    – TheTechGuy
    Mar 1, 2019 at 9:50
  • If yout just remove them, I think they remain in the index database and Google will try to re-visit them. But if they remain and fire the 410 header Google deindex them just after first revisit
    – Evgeniy
    Mar 1, 2019 at 10:07
  • I am thinking of setting `$header= ('HTTP/1.1 410 Gone'); and then echo that from search page and run the script. Temporarily it should remove these pages but will effect other pages too.. Well in fact I have made all search pages NoIndex as I probably should make google crawl this page. What do you say
    – TheTechGuy
    Mar 1, 2019 at 10:10
  • 1
    I would prefer 410 before noindex, which can be too slow and not mandatory
    – Evgeniy
    Mar 1, 2019 at 10:11
0

To remove indexed pages from SERP first you need to delete those pages and set redirection to the contact us page or which ever particular page might be.

After the site gets crawled again check that particular page appears on search results or not.

If the page still appears on search results you need to request Google to delete the page with the help of GSC. Steps to be followed to delete the pages which are appeared on search results. 1. Login to your webmaster tools. 2. Go back to the previous version of webmaster tools (i.e old version) 3. On the left-hand side, a menu will appear. 4. Click on Google Index and select remove URL's. 5. Copy & Paste the URL and follow the instructions. 6. After a few days you can check the status whether the page is deleted or not.

0

You may remove URLs from Google temporarily by using the Remove URLs feature the the Webmaster tools. As of this date (Aug. 2019), this feature is still in the old Webmaster tools and can be found on the following URL:

https://www.google.com/webmasters/tools/url-removal?hl=en&siteUrl=https://example.com

^ Replace example.com with your domain. From the menu, it is location under Google Index > Remove URLs

If there is a pattern in your URL, you can remove the entire directory by selecting that option in the dropdown. For your example PHP code, this would be search.php

enter image description here

I should also note: This isn't neccessarily an index issue - the index is a symptom of the primary issue, which is your wasted crawl budget.

By adding your scripting, you have drastically increased the number of pages needed to crawl by the Googlebot beyond your website's crawl budget. This really makes a mess of things and makes it harder for your other content to be more sticky in SERPs as they have a lower priority in your site hierarchy now due to the flood of irrelevant pages.

If you were to view your crawl stats, located on https://www.google.com/webmasters/tools/crawl-stats?hl=en&siteUrl=https://example.com (again, old version of webmaster tools not yet present in the newer search console), you will almost certainly identify a sharp spike in crawl when you added the script, but the spike will nowhere be enough to crawl the sheer volume of pages you created programatically.

Please add a noindex directive on all pages contains search.php to ensure the URL removal process will go smoothly, so if Google recrawls those pages that they will not reappear in the index.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.