1

I have a strange problem with my WordPress site that has no common solution.

For two months now in the middle of turning my site from a Lifestyle blog to a book-oriented blog and had to delete over 400 posts.

I got just 94 posts left now but the thing is that search engines are now indexing thousands of urls on my site. Google has hit over 23k urls while bling is at 1.4k.

After much research online I found that this issue is referred to as index bloat and its a common thing with eCommerce sites that delete a lot of content.

Solution? Using robot txt to disallow each single one of theses thousands of url. That kind of work is out of the question.

I ran a site:mysite.com search on google search and saw thousands of these urls. Its mainly my homepage and blog pages repeating themselves, over and over, thousands of times. I moved from blogger to Wordpress originally then successfully changed url structure from .html to /. Now all those dead urls have come alive, every url ever to exist on my site even those added by a malware a year ago which I cleaned up with 2 WordPress reinstalls. Thousands of URLs are now being indexed but traffic is zero!

It’s been over 2 weeks now and no change. What do I do? Get another domain or just wait it out?

Personally speaking, I would rather have a new domain cos the number of dead urls on this site now is shocking but my worry is that it might take google ages to index the new domain and give me traffic

Note: I also can't help wondering if my site has been hacked somehow. A strange email for Yandex Webmaster asking me to edit my settings when i had no account there only adds to this concern.

URL Pattern can be seen here. https://www.google.com/search?q=site:kingezesblog.com&rlz=1C1RLNS_enNG806NG806&ei=tpxpXOLpMteH1fAPtK-7kAQ&start=170&sa=N&ved=0ahUKEwjinJOcqMPgAhXXQxUIHbTXDkI4ggEQ8tMDCIEB&biw=1600&bih=789

|improve this question
  • What kind of URL pattern are you seeing, for example, with your home page that's indexed so many thousands of times? – Maximillian Laumeister Feb 16 at 2:38
  • Google doesn't index dead URLs. If the URLs return a 404 status, if the URLs redirect, or even if the URLs show some other page, Google won't index them. There has to be more to this. Robots.txt won't help you. Can you let us know a handful of the actual URLs? – Stephen Ostermiller♦ Feb 16 at 2:48
  • @ Maximillian Laumeister @ Stephen Ostermiller ... please follow this link to see the url pattern it a listing of them on google search .. google.com/… – Eze King Eke Feb 17 at 17:42
1

You recently switched your website's location from https://kingezesblog.com to https://www.kingezesblog.com. I can tell because your website answers to the www version of your domain, but the entries in google search link to the non-www version of your domain.

Before you added the www, your canonical URLs were incorrect because they were pointed at the www subdomain but none of your pages were served from the www subdomain. This likely caused Googlebot to ignore your canonical tags entirely.

Now that your website is hosted on the www subdomain and redirects have been put in properly, Google should recognize the now-valid canonical tag and begin to de-index the bloat in search. You should be done now.

Oh, and your robots.txt is preventing Google from discovering these changes, so you might want to remove some of the lines to make sure it can discover the new canonicals.

|improve this answer
  • No redirects are not really in place, its a redirection plugin in action but even that cannot seem to fix things. With your reference to robot txt, i just when in there snooping and found that my entire sitemap is in the non-www version of my site. In fact, the core of my site seems to be in the non-www version. I don't even know how to adjust my robot txt the way you advising to allow google in. Pls advice me. i am a novice – Eze King Eke Feb 18 at 0:44
  • @EzeKingEke I'm sorry to say this, but the scope of this question has expanded a lot. I wish you the best in fixing this, but the scope is outside the realm of what Stack Exchange can help with effectively, since Stack Exchange focuses on specific technical questions rather than site reviews. My advice is to hire a professional to look at your WordPress setup and do a full review of your website to make sure it's configured properly. – Maximillian Laumeister Feb 18 at 1:01
  • @EzeKingEke By the way, when I mentioned that your robots.txt was blocking Google, I meant specifically the Disallow: *?* and Disallow: /*?* rules. My gut says to remove those so that Google can discover the canonical tag on URLs that match those patterns, but I can't make any guarantees since I don't have specific knowledge of your website's configuration. Good luck, and again I recommend consulting with a professional, but if you have a specific technical question please feel free to post a new thread on Stack Exchange. – Maximillian Laumeister Feb 18 at 1:05
0

Google and Bing do not consider tags, categories, authors, brands or anything similar to be duplicate, thin content or bloatware. Pages that are useful for users should always be indexed, the only time you DO NOT WANT to index something is when its identical content, sensitive information or adds no value for users.

It takes time Winston, 2 weeks is not enough for Google to crawl 23K pages if it considers the site to have low authority, for example, Pro Webmasters is a high priority site because of the hundreds of thousands of visitors, content and links to it, a normal website that is under 1k - 25K with fewer links will receive a lot fewer resources which in turns means less time the Googlebot spends on your site every visit and how often it returns.

Some sites could take several months to remove 23K pages, if not longer, Google needs to return to those pages several times before it considers it a drop, but again, if these are useful pages, they should be indexed, if you have to many categories or similar, try and cut them down for your users, otherwise leave them indexed.

|improve this answer
  • Hi, Simon, i don't think you understand. Those 25k pages are worthless ghost URLs. All i have on my site right now is 94 posts so where in god's name is goggle getting the rest it is indexing? I took a good look at some of those urls when found them on search engine and they end like this... myste/page/2/dei_wordfence343565433566b343 ... it would seem that most of the URLS repeating themselves are from an old malware attack i had months back but removed via reinstalled WordPress and a fresh start via XML file – Eze King Eke Feb 17 at 17:38
  • xml sitemap has little to no infurence on Google removing URLs, you can an empty sitemap and Google will still index URLs it can crawls. You may be able to speed up the process by adding Disallow: /?wordfence* to your robots.txt file. – Simon Hayter♦ Feb 19 at 18:13
  • But yes, I misunderstood the question :) – Simon Hayter♦ Feb 19 at 18:14
0

Many of the URLs that Google is indexing have this pattern:

/?wordfence_lh=1&hid=B4942E0539C4AE55025B6DADC1652382

URLs like this are generated by the Wordfence security extension to WordPress. You should remove this extension, or at the very least turn of its "live traffic" feature that creates these URLs. Many other people have had this same problem with this plugin: 1, 2, 3, 4, 5, 7

Even after you fix your plugin, Google is never going to forget about all these weird URLs that were generated. The quickest way to solve this issue is going to be by telling Google that it should ignore the parameters.

  1. Open the URL Parameters Tool in Google Search Console
  2. Click "Configure URL parameters"
  3. Click "Add parameter"
  4. Add the parameter hid as "Doesn't affect page content"

Add a second parameter wordfence_lh the same way.

It may take a couple months, but all the pages should fall out of the Google search index.

|improve this answer
  • I fixed the Parameters. There are 2 more strange looking ones though, 'm' and 'filter_by' . .... Again, in regards to the WordFence extension and plugin, the damage was so bad that I had to delete the site and reinstall Wordpress then start from scratch with a single XML file. To the best of my knowledge, not even those wordfence urls are my site talk less of any wordFence extension. They have always been spam and 500 error generators – Eze King Eke Feb 18 at 11:16
  • if the m and filter_by parameters are similarly useless and don't change the page, then go ahead and add them in the Google Search Console URL Parameters tool too. – Stephen Ostermiller♦ Feb 18 at 11:18
  • I'm glad that you had already removed that plugin. As long as it was creating new URLs like this, your site was never going to have good SEO. Your site does have canonical URLs configured correctly, so these hid URLs will fall out of the Google index naturally as Googlebot gets around to crawling them. Using the parameters tool should be speed up the process. – Stephen Ostermiller♦ Feb 18 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.