0

I've just seen in Google Search Console that Google is doing 3 things with spammy internal search links. The link structure is always www.example.com/search?q=[spammy link here]

  1. Around 500 are green and indexed (they don't appear on the sitemap but are indexed in the overall page indexing)
  2. Around 300 are grey and blocked by robots.txt (the instruction here is disallow /search)
  3. 400 more are orange, indexed, though blocked by robots.txt

My robots.txt is set to disallow /search. Do you know why:

  1. these spam links are doing 3 different things
  2. will they impact my site negatively (I've seen some drop in visits since this spam started)
  3. Is there anything I can do about it?

Thank you, I've been Googling for days and still confused about a solution.

7
  • I think the best way is to add a robots meta tag / X-Robots-Tag http header, per developers.google.com/search/docs/crawling-indexing/…. This could be done either by changing the code generating the search result to add a <meta name="robots" content="noindex"> tag to the html file, or the server config files (say .htaccess) to add X-Robots-Tag: noindex to URLs starting with "search?q="
    – Uri Raz
    Apr 13, 2023 at 10:27
  • Thank you for your response. Do you know if there are any online guides on how to do either of these steps? My robots.txt file is un-editable due to the website builder I use.
    – Bias_
    Apr 13, 2023 at 10:48
  • Where does Google say it is finding links to these pages? I suspect that your site was hacked and that the hacker created links on your own site to them. Apr 13, 2023 at 13:22
  • How can any of them be green and indexed if you are disallowing /search in robots.txt? Are some of them using some other URL path? Apr 13, 2023 at 13:23
  • @StephenOstermiller "How can any of them be green and indexed if you are disallowing /search in robots.txt?" This is what has confused me - I don't know how they've indexed - every like has the same structure, something like this - www.example.com/search?q=naza8k-สล๊อค รับเครดิตเล่นฟรี 50🌟
    – Bias_
    Apr 13, 2023 at 13:59

1 Answer 1

-1

If your web server is Apache or LiteSpeed, you can add the X-Robots-Tag by adding the following to .htaccess

<FilesMatch "^search">
 Header always add X-Robots-Tag "noindex"
</FilesMatch>

I don't know what software implements the search functionality for your website, so I can't tell how to change it to add a robots meta tag.

1
  • Sadly this is not accurate. Search needs to be blocked by robots.txt, not noindex tags, to prevent Google from wasting resources by crawling the infinite space. Noindex tags only prevent indexing but not crawling, so Google will continue to crawl them in they are not added to robots.txt. If they are in robots.txt however, Google will not see the noindex tag anyway, so it’s better to only use robots.txt for this and not bother with noindex. May 10 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.