2

Why has GoogleBot started last Friday to do POST-requests on a page. I can see in the logfile (just an example, had about 10.000 entries over the weekend - the url in the log is changed):

66.249.79.55 - - [15/Jul/2019:08:34:53 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:35:06 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:35:18 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:35:20 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:35:34 +0000] "POST /Contact/ HTTP/1.1" 200 16731 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:35:45 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:35:46 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:36:00 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:36:10 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Would setting the page to index/nofollow help in stopping this behaviour?

Funny side node: that page is using Google recaptcha V3, so in the google recaptcha stats I can see about 95% of bad requests (looks like they are from Googlebot itself).

  • Tried the index/nofollow approach wich doesn't help. Using the robots.txt to disallow the page does help - but it would be preferrable to have it in the search results. Is there any other way to achive this? – Seb Aug 13 at 12:25
1

Yes, GoogleBot is capable and does send POST requests (if it is safe - according to Google) - details here.

In the same article, it talks about blocking the URL via robots.txt - which is a fair approach, given your contact URL isn't in any way a secret and therefore safe to reference in robots.txt (and block that way). Couldn't confirm whether or not a no-follow will prevent the post request - from my experience "no-follow" is a recommendation rather than a policy.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.