2

How to stop indexing search result using Robots.txt?

hello.

Recently, I'm having an issue with indexed pages in Google Search Console growing 10x.

These pages are mostly search query (/?s=) and consist of spam queries that have nothing to do with my website, such as Casino and Porn.

What I don't understand is that I've been using robots.txt like this about 5 years ago.

Sitemap: https://example.com/sitemap_index.xml
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Disallow: *?orderby=*
Disallow: /?cat=usd&order=desc
Disallow: *?cat=usd&order=desc
Disallow: /?s=
Disallow: /search/
Disallow: /*/page/*
Disallow: */page/*
Disallow: /?s
Disallow: */uncategorized/
Disallow: */?s
Disallow: *?s=
Disallow: /test/*
Disallow: /test2/*
Disallow: /test3/*
Disallow: /test4/*
Disallow: /test5/*
Disallow: /test/
Disallow: /test2/
Disallow: /test3/
Disallow: /test4/
Disallow: /test5/

Nevertheless, the search results were indexed by Google.

Could someone please let me know if there is a problem with the Robots.txt I use?

Improve this question
19
  • 2
    It is likely that your site was hacked. The hackers may be surreptitiously using a different robots.txt file for Googlebot. Have you cleaned up the hack? Have you tested the indexed URLs against Google's robots.txt Tester?
    – Stephen Ostermiller ♦
    Sep 4, 2022 at 10:29
  • 1
    Is it possible that the page with such a param returns a different canonical url to one without a param ?
    – Rohit Gupta
    Sep 4, 2022 at 12:37
  • 1
    To add to the previous two good suggestions, how do you check that your search results are being indexed? It is usually pretty hard to make google index search results. And what does your site show for those search results? I typically make my enterprise clients serve a custom 404 page when nothing is found for a query exactly for this reason.
    – BNazaruk
    Sep 4, 2022 at 17:32
  • Hi, @StephenOstermiller Thanks for answering. I also suspected malware firstly and checked my DB and whole website but there was no problem. Checking Robots.txt is also fine.
    – cheonmu
    Sep 7, 2022 at 9:31
  • 1
    I still think your site is hacked, or had been hacked at some point. Its possible that Google is just indexing search results based on external links. But with thousands of URLs, it seems way more likely to be a problem with your site itself.
    – Stephen Ostermiller ♦
    Sep 7, 2022 at 9:51

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.