2

Google is indexing all kinds of spammed internal search results from my wordpress site, and I can't figure out how to put a stop to it. I added Disallow: /search/ to my robots.txt file, but that didn't stop google. They still index them, but now they say "Indexed, though blocked by robots.txt"

What can I do to stop these from being indexed? And how do they even show up for google? Is there somewhere within wordpress that these are being stored that google is picking them up from?

Any help would be much appreciated!

Here's what I see in google search console:

enter image description here

2
  • When you go to those pages are they returning a 404 (or better, a 410) respobse? Sending those error codes will tell Google to remove the results from the index - apparently the 419 is slightly better.
    – davidgo
    Jul 7 at 0:06
  • 1
    Google can index (based on links) despite being blocked by robots. Possible duplicate of webmasters.stackexchange.com/questions/117744/…
    – Kannan
    Jul 7 at 3:02
2

You also need to add noindex meta tag in the template page where your search result is dynamically rendered.

<meta name="robots" content="noindex" />

Usually, a CMS like WordPress uses a template page that dynamically renders the search result in your website. You need to add that no index meta tag in to that template page. However, please make sure that template is only for the search result pages to not affect other pages.

1
  • If you add the no index tag to the search results you need to remove the disallow from robots.txt. when search results are disallowed in robots that text Google won't crawl them to see the new index tag. Google can't know about the no index bag unless it can crawl the URLs. Jul 10 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.