6

Why has GoogleBot started last Friday to do POST-requests on a page. I can see in the logfile (just an example, had about 10.000 entries over the weekend - the url in the log is changed):

66.249.79.55 - - [15/Jul/2019:08:34:53 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:35:06 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:35:18 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:35:20 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:35:34 +0000] "POST /Contact/ HTTP/1.1" 200 16731 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:35:45 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:35:46 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:36:00 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.55 - - [15/Jul/2019:08:36:10 +0000] "POST /Contact/ HTTP/1.1" 200 16730 "https://www.example.com/Contact/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Would setting the page to index/nofollow help in stopping this behaviour?

Funny side node: that page is using Google recaptcha V3, so in the google recaptcha stats I can see about 95% of bad requests (looks like they are from Googlebot itself).

1
  • Tried the index/nofollow approach wich doesn't help. Using the robots.txt to disallow the page does help - but it would be preferrable to have it in the search results. Is there any other way to achive this?
    – Seb
    Aug 13, 2019 at 12:25

2 Answers 2

5

Yes, GoogleBot is capable and does send POST requests (if it is safe - according to Google) - details here.

In the same article, it talks about blocking the URL via robots.txt - which is a fair approach, given your contact URL isn't in any way a secret and therefore safe to reference in robots.txt (and block that way). Couldn't confirm whether or not a no-follow will prevent the post request - from my experience "no-follow" is a recommendation rather than a policy.

0

I also had a lot of Googlebot's POST requests in my nginx access log. With a ratio of 10:1 to GET requests. Those POST requests are not necessary for indexing. They have shorter URL than GET requests:

POST /Contact/
GET  /Contact/lang/en/details.html

I have prohibited bots from sending POST requests using the following directive in robots.txt:

Disallow: /Contact/$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.