My website has been scammed using some "scamming-web-site steals my content through a proxy and serves the stolen content from their own domain and replaces my site in the SERP"-technique.

The Apache log tells me the scam started with the following entries: - - [14/Apr/2017:05:57:32 +0200] "GET / HTTP/1.1" 200 27428 "https://dbwebb.se/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" - - [14/Apr/2017:19:55:46 +0200] "GET / HTTP/1.1" 200 27428 "https://dbwebb.se/" "AppEngine-Google; (+http://code.google.com/appengine; appid: s~gce-spider)" - - [14/Apr/2017:19:55:47 +0200] "GET / HTTP/1.1" 200 24445 "https://dbwebb.se/" "AppEngine-Google; (+http://code.google.com/appengine; appid: s~gce-spider)" - - [14/Apr/2017:19:55:50 +0200] "GET / HTTP/1.1" 200 27428 "https://dbwebb.se/" "AppEngine-Google; (+http://code.google.com/appengine; appid: s~gce-spider)"

It seems the scam originated from the User-Agent:

"AppEngine-Google; (+http://code.google.com/appengine; appid: s~gce-spider)"

That same User-Agent did then visit my website each 14 days up til June 23 2017.

Is this User-Agent containing gce-spider a well known scammer, a really bad bot?

  • If nobody has posted about the spider on the internet already, I don't know of any other method for discovering information about it. It is always possible that it is a custom user agent that is used just for scraping your site. I'd imagine that the user agent would be configurable in scraper software. – Stephen Ostermiller Aug 9 '17 at 9:10
  • The important part is AppEngine-Google. I block all accesses by either this string or from the IP address blocks automatically by default. Ever since Google introduced the AppEngine all matter of bot hell has rained down upon upon my website. None of it was worth a damn. So I block it all. – closetnoc Aug 9 '17 at 15:45

The first part of those user agents:

AppEngine-Google; (+http://code.google.com/appengine; appid

indicate that it's some kind of software running on Google App Engine. It's a common user agent.

This is a common user-agent fragment which Google prepends to all apps. The next bit which follows is the individual "app name" that a developer has chosen.

In the case of the bot that's hitting your server and stealing your content, it's: gce-spider (sometimes the app names have a s~ in front of them too)

There's not much more than can be gleaned from looking at the user agent.

Searching around, it's been around since at least 2016. If you're sure it's stealing content, you should report it to Google, particularly considering it's using "GCE" in it's app name (Google Compute Engine) - to me, that should be a reserved word! (in the way that you can't create a Facebook app with the word "Facebook" in it because it appears misleading)

| improve this answer | |
  • Please be careful about linking to or promoting your website in your answers. – John Conde Aug 9 '17 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.