1

I have the search form on a section of a site that is getting bombarded with requests like:

search query = baccarat philippe starck furniture, ➡️g188.ph⬅️, play free slots with bonus gcash logo cash, e sabong gcash slot games, ➡️g188.ph⬅️, baccarat bonus rules philippines islands countvv

or

search query = 手机上查找身份证信息怎么查-(⏭查询网站gps67.com⏮)-中国移动查自己身份证信息-根据身份证号查信用信息-利用身份证号查车次信息-(⏭查询网站gps67.com⏮)-执法上门查身份证信息NQ  

So quote obviously junk spam stuff.

However, the REMOTE_ADDR of the request is 66.249.64.244 which traces back to Google.

The HTTP_USER_AGENT is:

"HTTP_USER_AGENT":"Mozilla\/5.0 (Linux; Android 6.0.1; Nexus 5X Build\/MMB29P) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/110.0.5481.100 Mobile Safari\/537.36 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)",

The fact it's reading Android 6 on a Nexus 5 raises some suspicions... but, could a request have a spoofed REMOTE_ADDR? I don't see how, so is the only feasible explanation that this is Google, and somewhere they have found and are following links to my search form, e.g.

example.com/search.php?keyword=spam

and just crawling these external links?

But I'm also not sure about this theory, as the HTTP_REFERER is internal, e.g. from example.com/search.php?keyword=otherspam

I can't just put the IP into my firewall ban list, as it is a genuine IP of Google's bots.

I can write some on-site logic to try and ban this particular user but again, not ideal as it's Google's IP!

FYI my traffic is routed through Cloudflare.

Any ideas or suggestions?!

Thanks

Full $_SERVER below:

{
"HTTP_HOST":"www.example.com",
"HTTP_USER_AGENT":"Mozilla\/5.0 (Linux; Android 6.0.1; Nexus 5X Build\/MMB29P) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/110.0.5481.100 Mobile Safari\/537.36 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)",
"CONTENT_LENGTH":"320",
"HTTP_ACCEPT":null,
"HTTP_ACCEPT_ENCODING":"gzip",
"HTTP_ACCEPT_LANGUAGE":"en-US",
"HTTP_CDN_LOOP":"cloudflare",
"HTTP_CF_IPCOUNTRY":"US",
"HTTP_CF_RAY":"79e3e89f1c3e82ba-IAD",
"HTTP_CF_VISITOR":"{\"scheme\":\"https\"}",
"CONTENT_TYPE":"application\/x-www-form-urlencoded; charset=UTF-8",
"HTTP_COOKIE":"PHPSESSID=8c68cbd41264f9c72bd863bb3d53135a; web_4f857_lastUrl=https%3A%2F%2Fexample.com%2FshopSearch.php%3Fkeyword%3D%26keyword%3Dsabong%2520slots%2520capital%2520bonus%2C%2520%25E2%259E%25A1%25EF%25B8%258Fg188.ph%25E2%25AC%2585%25EF%25B8%258Fsabong%2520slots%2520capital%2520bonus%2C%2520-free%2520slots%2520with%2520bonus%2520coins%2520gcash%2520download%2520in%2520laptop%2C%2520%25E2%259E%25A1%25EF%25B8%258Fg188.ph%25E2%25AC%2585%25EF%25B8%258Fsabong%2520slots%2520capital%2520bonus%2C%2520-free%2520slots%2520with%2520bonus%2520coins%2520gcash%2520download%2520in%2520laptop%2Clm; sticky_=http:\/\/10.0.1.113:80",
"HTTP_FROM":"googlebot(at)googlebot.com",
"HTTP_ORIGIN":"https:\/\/www.example.com",
"HTTP_REFERER":"https:\/\/www.example.com\/shop\/search\/?keyword=sabong%20slots%20capital%20bonus,%20%E2%9E%A1%EF%B8%8Fg188.ph%E2%AC%85%EF%B8%8Fsabong%20slots%20capital%20bonus,%20-free%20slots%20with%20bonus%20coins%20gcash%20download%20in%20laptop,%20%E2%9E%A1%EF%B8%8Fg188.ph%E2%AC%85%EF%B8%8Fsabong%20slots%20capital%20bonus,%20-free%20slots%20with%20bonus%20coins%20gcash%20download%20in%20laptop,lm",
"HTTP_X_FORWARDED_FOR":"172.70.175.103",
"HTTP_X_FORWARDED_HOST":"www.example.com",
"HTTP_X_FORWARDED_PORT":"443",
"HTTP_X_FORWARDED_PROTO":"https",
"HTTP_X_FORWARDED_SERVER":"94ac1df2b1c6",
"HTTP_X_REAL_IP":"172.70.175.103",
"HTTP_X_REQUESTED_WITH":"XMLHttpRequest",
"PATH":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin",
"SERVER_SIGNATURE":" Apache\/2.4.53 (Debian) Server at www.example.com Port 80<\/address>\n",
"SERVER_SOFTWARE":"Apache\/2.4.53 (Debian)",
"SERVER_NAME":"www.example.com",
"SERVER_ADDR":"10.0.1.22",
"SERVER_PORT":"80",
"REMOTE_ADDR":"66.249.64.244",
"DOCUMENT_ROOT":"\/var\/www\/html",
"REQUEST_SCHEME":"http",
"CONTEXT_PREFIX":"",
"CONTEXT_DOCUMENT_ROOT":"\/var\/www\/html",
"SERVER_ADMIN":"webmaster@localhost",
"SCRIPT_FILENAME":"\/search.php",
"REMOTE_PORT":"44798",
"GATEWAY_INTERFACE":"CGI\/1.1",
"SERVER_PROTOCOL":"HTTP\/1.1",
"REQUEST_METHOD":"POST",
"QUERY_STRING":"",
"REQUEST_URI":"\/search.php",
"SCRIPT_NAME":"\/search.php",
"PHP_SELF":"\/search.php",
"REQUEST_TIME_FLOAT":1677196697.564587,
"REQUEST_TIME":1677196697,
"argv":[
],
"argc":0,
"PHP_SELF_ORIGINAL":"\/search.php"
}

UPDATE: I have followed https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot to vefify the IP is a genuie Google Bot, and it is...

So is the best explaination that somehow somewhere links to example.com/search.php?search=spam exist, and Google has found and is crawling them?

1
  • Correct, this is quite common. Commented Feb 24, 2023 at 10:29

1 Answer 1

2

You should disallow your site search in robots.txt:

User-Agent: *
Disallow /search.php

Whenever you have an infinite URL space like this, you don't want Googlebot to try to crawl it. It can generate a lot of crawling for something that won't help your SEO.

Google doesn't want to index site search results. They view site search pages as low quality content. They say it is very poor user experience when a user clicks from Google only to find themselves on another similar page of search results. Google sometimes penalizes sites that allow their site search to be crawled and indexed because of that. See Matt Cutts: Search results in search results.

There are three ways that Googlebot finds these URLs:

  1. It finds links to them from some other site.
  2. There is malware installed on your site that is creating pages only visible to Googlebot.
  3. Googlebot tests search terms on its own to test if you create pages for spammy searches (like "viagra") so that it knows whether or not it should penalize your site.

Using robots.txt to block these URLs will help in all those cases. It would also be a good idea to check Google Search Console and make sure you recognize all the URLs that are getting indexed. If you see spammy URLs that you didn't create, you will need to clean up the malware that got installed on your site.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.