2

I am getting spammed by this IP: 66.249.79.70, and this website is telling me that it is from Google: https://ipinfo.io/66.249.79.70

It is filling one of my input boxes and keeps posting. I want to block this bot (with IP address: 66.249.79.xxx) from accessing one specific page (e.g., www.example.com/blocked.php) using an .htaccess file, how can I do that? I don't want to block the IP address completely though because I don't doubt it's from Google.

|improve this question
5

Since this does appear to be the real Googlebot, the recommended way to block access/crawling is to use /robots.txt:

User-agent: googlebot
Disallow: /blocked.php

However, if you still want to block this IP using .htaccess then you can do something like the following, near the top of your root .htaccess file:

RewriteEngine On
RewriteCond %{REMOTE_ADDR} =66.249.79.70
RewriteRule ^blocked\.php - [F]

This will return a "403 Forbidden" when a user with the above IP address attempts to access /blocked.php. Note that the "user/bot" still tries to access the URL by making a request, whereas with robots.txt they shouldn't even make the request (assuming they obey the robots.txt "standard"; which Googlebot does).

However, bear in mind that Googlebot can crawl from different IP addresses, so this may need to be updated in the future.

If you did want to block the range of IP addresses denoted by 66.249.79.xxx, then you could change the above condition to:

RewriteCond %{REMOTE_ADDR} ^66\.249\.79\.

However, that could block more than just Googlebot and may not be any more successful at blocking Googlebot than checking the specific IP address above (Googlebot does not necessarily crawl on a continuous IP block).

|improve this answer
  • 2
    The simple and efficient method is block it via robots.txt. .htaccess cause many problem as you said, and X robots header tag(like nofollow and noindex) does not stop Googlebot completely to crawl. Might setting up crawl rate in search console helps a little bit. Another thing is that OP should use some kind of capcha code to avoid auto fill up by Googlebot. – Goyllo Jan 10 '17 at 9:23
  • "User-agent: googlebot Disallow: /blocked.php " If i use this then google bot will not stop to crawl entire website right? – arunwebber Jan 10 '17 at 9:37
  • Yes, that's right. Bots will crawl everything by default, you only need to include the URLs you want to block. – MrWhite Jan 10 '17 at 9:39
  • If i use this "User-agent: googlebot Disallow: /blocked.php " when google bot visits blocked.php what it will see? – arunwebber Jan 10 '17 at 9:45
  • Google will read and process your /robots.txt file (this can take a day or so - check in Google Search Console on the status). Once Google has processed your /robots.txt file then Googlebot simply won't visit /blocked.php at all. robots.txt blocks crawling. – MrWhite Jan 10 '17 at 9:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.