Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.
Sign up to join this community
I am getting spammed by this IP: 66.249.79.70, and this website is telling me that it is from Google: https://ipinfo.io/66.249.79.70
It is filling one of my input boxes and keeps posting. I want to block this bot (with IP address: 66.249.79.xxx) from accessing one specific page (e.g., www.example.com/blocked.php) using an .htaccess file, how can I do that? I don't want to block the IP address completely though because I don't doubt it's from Google.
Since this does appear to be the real Googlebot, the recommended way to block access/crawling is to use /robots.txt:
User-agent: googlebot
Disallow: /blocked.php
However, if you still want to block this IP using .htaccess then you can do something like the following, near the top of your root .htaccess file:
RewriteEngine On
RewriteCond %{REMOTE_ADDR} =66.249.79.70
RewriteRule ^blocked\.php - [F]
This will return a "403 Forbidden" when a user with the above IP address attempts to access /blocked.php. Note that the "user/bot" still tries to access the URL by making a request, whereas with robots.txt they shouldn't even make the request (assuming they obey the robots.txt "standard"; which Googlebot does).
However, bear in mind that Googlebot can crawl from different IP addresses, so this may need to be updated in the future.
If you did want to block the range of IP addresses denoted by 66.249.79.xxx, then you could change the above condition to:
RewriteCond %{REMOTE_ADDR} ^66\.249\.79\.
However, that could block more than just Googlebot and may not be any more successful at blocking Googlebot than checking the specific IP address above (Googlebot does not necessarily crawl on a continuous IP block).
.htaccess cause many problem as you said, and X robots header tag(like nofollow and noindex) does not stop Googlebot completely to crawl. Might setting up crawl rate in search console helps a little bit. Another thing is that OP should use some kind of capcha code to avoid auto fill up by Googlebot.
– Goyllo
Jan 10 '17 at 9:23
/robots.txt file (this can take a day or so - check in Google Search Console on the status). Once Google has processed your /robots.txt file then Googlebot simply won't visit /blocked.php at all. robots.txt blocks crawling.
– MrWhite
Jan 10 '17 at 9:51
