3

My WordPress site, not that much huge content. Recent faced problem of High CPU bandwidth usage. Within seconds it gets 100% and the server become down. After a huge analysis, I found that in Google Search Console their indexed status about 2,64,023. And the URL Parameters where URLs monitored over 24,981,662 for individual parameter. That is insane. Used filtering option. After find out the problem noindex those option from Yoast plugin, Edit the parameters. But no change in index status. It's increasing day by day. So I want to no-index those parameters permanently from Google Search Console and also want to delete it. How can I do that through .htaccess file? That will surely decrease the Total indexed number from Google Search Console.

Here is the indexed URL parameters from Google Search Console:

enter image description here

  • Are these URL parameters important for the function of your website? Or do you simply want to discard them? – DocRoot Nov 27 '17 at 11:41
  • No it's not important. Are those Filtered Parameters causing the High bandwidth usage. if i delete those, indexed status decrease or not? I am really worried...what should i do actually..... – Mourin Nov 27 '17 at 12:20
1

It looks like you should probably be blocking these URLs (with URL parameters) in your robots.txt file, to prevent search engine bots (ie. Googlebot) from crawling these URLs in the first place. For example, to block all URLs with query strings:

User-agent: *
Disallow: /*?

Within Google Search Console (formerly Webmaster Tools) you can also explicitly tell Google how to handle each URL parameter. Under Crawl > URL Parameters. For example, your filter_display parameter might be defined as:

  • Does this parameter change page content seen by the user?
    "Yes: Changes, reorders or narrows page content"
  • How does this parameter affect page content?
    "Narrows"
  • Which URLs with this parameter should Googlebot crawl?
    "No URLs" (or perhaps "Let Googlebot decide" if you trust Google, given the previous options)

How can i do that through .htaccess file?

You mentioned in comments that these URL parameters are "not important". However, they do look like they provide some user features (eg. filtering, sorting, ...)? In which case, you probably don't want to use .htaccess. Using .htaccess you could canonicalise the URL and redirect URLs with these URL parameters. This would completely remove these URL parameters from your site - which could even break your site functionality?


UPDATE: Your robots.txt file (copied from comments):

User-agent: *
Disallow: /*?

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

User-agent: *
Allow: /wp-content/uploads/
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
Disallow: /images/
Disallow: /wp-content/
Disallow: /index.php
Disallow: /wp-login.php

This would not work as intended. You have conflicting groups. ie. Three groups that all match User-agent: *. Bots only process one block of rules. The block that matches is the one that matches the "most specific" User-agent. The User-agent: * block matches any bots that didn't match any other block. From these rules Googlebot will simply crawl everything (unrestricted), including all your URL parameters - if this is causing problems for your server (as you suggest) then this is not what you want. And from these rules I would "guess" that all other bots will match the first User-agent: *

(But, even if you adopted different reasoning and assumed multiple blocks could be processed, this wouldn't make sense...?)

Depending on your requirements, this should be written something like:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /wp-content/uploads/
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
Disallow: /images/
Disallow: /wp-content/
Disallow: /index.php
Disallow: /wp-login.php
Disallow: /*?

I assume that, if this is a WordPress site, then you don't want even Googlebot to crawl everywhere?

From these rules, all other (good) bots are prevented from crawling your site.

  • Thank you For your Reply. Now the main Issue is to Decrease The indexed URL. My Blog site not much big. But the google index status shows 264,473 no of index. It's increasing day by day. Also the Server CPU usage getting higher, all time greater than 80% and the server being un reached. While analysis found here support.google.com/webmasters/answer/76401?hl=en that excess url parameters issue for the excessive index. So thought if i delete those parameters may be the index number will decrease. So what can be the solution to minimize the index number, and to stop it....Please help... – Mourin Nov 30 '17 at 5:41
  • Implement the robots.txt method as mentioned above (and set the URL parameters in GSC) - that should reduce the crawl rate and prevent the increase in the index status. Confirm this is working OK in Google Search Console (and test some URLs). However, it can take a day for Google to pick up the robots.txt file. However, the methods mentioned above prevent crawling (and hence indexing of new URLs) which appears to be your main problem). They won't necessarily reduce the number of already indexed URLs. – DocRoot Nov 30 '17 at 10:04
  • Thank you. I set the parameters in GSC, but set parameter affect pag content set as "Sorts". After two weeks i don't find any changes in GSC, but the headache is index stutus still increasing and the server CPU usage also remain high all time. So if i use the above 2 lines in robots.txt what will be the benefit/changes? – Mourin Dec 3 '17 at 9:04
  • As mentioned above, those two lines in robots.txt prevent URLs with query strings from being crawled by Google (and any good bots). If they aren't crawled then no CPU usage. Whilst this doesn't necessarily prevent URLs from being indexed (particularly if they are already indexed), it certainly dramatically reduces the chances of the URLs being indexed and returned in the SERPs. – DocRoot Dec 3 '17 at 16:39
  • Thank you for your comment. i added this in this way at robots.txt file User-agent: * Disallow: /*? User-agent: * Disallow: / User-agent: Googlebot Disallow: User-agent: * Allow: /wp-content/uploads/ Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /junk/ Disallow: /images/ Disallow: /wp-content/ Disallow: /index.php Disallow: /wp-login.php hope it will reduce the CPU usage and prevent the URL's being indexed. Please check whether the robots.txt file okay or not... – Mourin Dec 4 '17 at 6:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.