Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I had a bot of some sort crawling my site like crazy for the last day or so. It says in Google Analytics that it's from San Jose in the US. The IP addresses that appeared in the access logs were,, I blocked each of those and then it stopped for a couple of hours. Now it's back and in the access logs, it says that the domain accessing the site is my own, i.e. www.example.ie

How can I stop this? I blocked the whole range of 38.99.82 by placing the following in a htaccess file.

# Block Site Crawling our website #
Order Deny,Allow
Deny from,,,
share|improve this question

closed as unclear what you're asking by Stephen Ostermiller Dec 13 '13 at 12:21

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question.If this question can be reworded to fit the rules in the help center, please edit the question.

Bots don't show up in Google Analytics. Are you sure they are bots? – Brendon Dec 12 '13 at 23:46
Hi Brendon, I'm not sure what it is tbh. At any one time, there are between 8 - 12 pageviews from either of those 3 IP addresses. It's been pretty constant for the last two days. Appreciate the help. – user23744 Dec 12 '13 at 23:47
"the domain accessing the site is my own" - the HTTP Referer? This can be easily faked so take it with a pinch of salt. – w3dk Dec 13 '13 at 9:55
If you are blocking the bot by IP address, how are they accessing the site? You may still see hits in your logs, but those should be "access denied" hits (which are cheap and your server should be able to handle tons of.) – Stephen Ostermiller Dec 13 '13 at 12:18
I have a reverse proxy server setup and a webserver. I had only looked at the logs on the webserver. When I looked at the logs on the reverse proxy, I was able to find the other IP address and block it. I got some help off my host this morning which will help me fix the rest. Thanks for your help guys. – user23744 Dec 13 '13 at 13:43