41

Google has no way of knowing when you hit your own site with your own bot. Google can't see anything that happens on your server. Even if you use Google Analytics, your bot won't show up in it. Google Analytics relies on client side JavaScript to record hits. Most bots don't execute JavaScript. If your bot does execute JS, you should be able to ...


15

Google doesn't care about your traffic. Even if you use Google Analytic, Google never uses that data for search engine ranking algorythm. Gary Illyes from Google says in a tweet We don't use anything from Google analytics in the "algo".


11

They search for and fill in every form they can find. They're looking for any vulnerability that they can exploit for their gain. It might be to see if they can gain access to your site or web server. It can be to compromise your form to send out spam. It's worth their time to do because it's all automated. They just set their bots free and let them go ...


11

This is a Joomla 0 Day Attack. Information found here: https://blog.sucuri.net/2015/12/remote-command-execution-vulnerability-in-joomla.html This is not a vulnerability test despite the __test. It is an attack. Make sure that any Joomla install is as up to date as possible. Another option is to simply use .htaccess to intercept this exploit by looking for ...


6

I'm the author of the free open source Ostermiller Contact Form which is designed to thwart spammers. I have found that the biggest reason that spammers fill in your contact form is link spam. To many spam bots, you contact form looks little different than a guest book form or a form to comment on a blog. My contact form software has rules that ...


4

The IP address that you linked does not resolve to a Google hostname therefore it is not Google. The person or bot is scanning your site for vulnerabilities. The first one is attempting to find a Joomla vulnerability. These events are a regular occurrence on most websites, You should ensure that you are following best practices and harden your website, the ...


3

This could be anything. It doesn't look much like an attack (although it could be an attempt to pollute your search results - but it's a 404 anyway so no worries there, although it's polluting your reports). Note that if GSC is reporting these 404s then it must have found and crawled these URLs from somewhere, either through a misconfiguration on your site ...


3

Since you use tags that start with web, I assume you are building a system that requires HTML to be downloaded in order for a paid impression to work. What you need to do is learn robot behaviour by looking at the server access log files. On a server with apache installed, the file is typically named access_log. If your server is used frequently, you will ...


3

The space is a delimiter (ie. a special character) in .htaccess so must be backslash escaped if you want to match a literal space in the regex. Eg. DV\ CRAWLER. (Otherwise you are likely to get a less than helpful 500 Internal Server error.) Or, you can use the shorthand character class \s which matches any white space character (space, tab or new line / ...


3

I realize that this is an old post but after researching this I came upon an article explaining the purpose of this user agent. In this article they explain that this is a user agent string used to fetch images in real time when being accessed through the Bing app. Meaning as you scroll through pictures within this app you may end up clicking on one to ...


3

I think 404 does make sense in some ways. If an undesirable walks up to your house, sure you could shout at them to leave, but it might make more sense just to not answer the door. This has the added bonus of them not knowing whether you're actually not home or just not answering. A 404 is the equivalent of not answering the door. Plus it's semantically ...


3

Google won't know about traffic to your website unless the 'bot (or human user's web browser) does TWO things: Google Analytics asked you to install some Javascript on your web pages. The browser or bot must actually execute that Javascript code. The Javascript code will ask the browser to send an HTTP request to a Google website. The browser or bot must ...


2

The IP address you provided is a subscriber IP with the domain name of ip16-90-210-87.adsl2.static.versatel.nl. This is a person at home or in a small office and would not indicate valid bot activity as it normally appears. Here are some of the particulars: AS13127 - Tele 2 Nederland B.V. IP Address Range: 87.208.0.0 - 87.215.255.255 NetMask: Block: 87....


2

Looks like your garden variety web scraper. Some website owner who lacks ethics is scraping your website.


2

It looks like that code was developed to stop bots probing humans.txt via query strings. Not Blocked: http://example.com/humans.txt Blocked: http://example.com/?some_path=http://www.google.com/humans.txt? There is plenty of online guides about blocking humans.txt additionally lots of websites explaining what does what, a lot of those rules in that block ...


2

A few more methods of bot traffic detecting: Verification of user agent (it is a client application that uses a particular network protocol). Looking for a highly specific match to something like a malware signature or specific executable or C&C connection address. Examining such behavioral parameters as a depth of view, the duration of visit, ...


2

Any website, or any server that sends a huge volume of emails runs the risk of their IP or domain being added to email blacklists. Also, similar, or exact emails are higher on the risk factor, especially when it comes to 'MASS' public email addresses such as: @yahoo.com @gmail.com @hotmail.com @aol.com You can find out if your server IP or domain has been ...


2

In regard to your automated emails failing, these spammer email addresses used for the fake registrations often bounce because they either don't exist or quickly become full, or are flagged as spammer addresses - for this reason your server will have a high rate of bounced emails. This will affect the mail reputation of your server and will in turn degrade ...


2

I'd recommend: connection: close on pages that don't list any other resources that are stored on your server for best performance. If you can setup a 404 error page without loading any images or special assets, then connection: close is good. If you need resources to completely load the error page, then connection: close isn't as good since the ...


2

Mystery possibly solved? Google pushed out Panda 4.2 on July 18 2015, however, prior to that there was a Quality Update (also called Phantom 2) that some say is related to the Panda roll-out. Clues to hits are fluctuations as some call them tremors and spikes. Here is a quote from: http://www.thesempost.com/google-panda-update-everything-we-know-about-...


2

Hm... The quick way to prevent robots visiting your site is put these two lines into the /robots.txt file on your server: User-agent: * Disallow: / Try to put this for one day, and see what happens. This only helps with well-behaved robots. But, if you like to disallow that page specifically, Disallow: /featured/ Do that, and see what happens.


2

First step is to report the spam to Google, additionally you can do a whois search on the domains and report the spam to the registrar, most registrars have anti spam policies.


2

Additionally to other answers, note that the fact that this attack apparently worked suggests you are running an old, insecure version of PHP. A fix for the bug that this attack exploits was released in september 2015. Run your update process and make sure it pulls in the most recent version of PHP. And check for other outdated programs that are Internet-...


2

These are all used in tracking URLs. A SubID is a string of alphanumeric characters generated at the end of a redirect URL, which records a user-defined variable. uid is Unique Identifier. Check if you have some affiliate program


2

To answer your question, no, I don't think adding that many lines in htaccess etc. is a good solution. As you suspected you only block already detected bots. Its more efficient to use a list like spamcop https://www.spamcop.net/fom-serve/cache/291.html and Apache Spamassasin https://spamassassin.apache.org/ directly on mailserver level to protect you from ...


2

These are 2 methods I use. There may be other better ones but these are 2 quick ones that I've used successfully. Method 1 You could enable 2 Factor Authentication. This is a quick solution for such instances when you're under brute force attacks. Right now I know of 2 options to do this: If you're already using Jetpack, you can enable this option from ...


1

If it is a small/middle sized company with limited bandwidth (<1-10 Gbps) changing IP address pool is the best, cheaper and quicker option. Only carriers, CDN providers or big companies can mitigate DoS attacks with other solutions (and the bandwidth is not a problem for them). About DNS propagation, whois registers are the one that takes a very long ...


1

You have a couple of options here that may help: 1) Lock out very old browsers and user agents that don't make sense. 2) Load the form with ajax after JS detects mouse/tap movement and confirms against server side session token. 3) Use a better captcha such as reCaptcha. 4) Use a throttler at a decent rate such as 8-10 requests every 5 secs get blocked ...


1

Find out how many videos viewed by normal users per hour. Let's assume It is 10 videos viewed by most of visitors per hour. Store & use the # of videos viewed using session + cookie + IP address. If the number of videos viewed are higher than 10 for any session, then use simple captcha to avoid the bots. If the captcha verification is failed ( ...


1

When in doubt, add parenthesis and escaping to regular expressions. Try this first: (Baiduspider|(DV CRAWLER)) I think that your problem is that it evaluating as "Baiduspider or DV followed by CRAWLER" when you don't have the paranthesis. If that doesn't work, then try escaping the space: (Baiduspider|(DV\sCRAWLER)) Where \s is any white space ...


Only top voted, non community-wiki answers of a minimum length are eligible