6

2 weeks ago I found out that some domains were proxying or scraping my site. So when you accessed theirsite.com/some/parameter they would render exactly the same as mysite.com/some/parameter but replacing all instances of mysite.com with theirsite.com, removing all javascript code, and adding before the tag some liveinternet.ru widget or something like that. There were like 10 domains doing it and I didn't see it before, so google indexed those pages and there were so many of those sites that they outranked me, propably even my links were deleted from search results. I managed to get IP's of servers running on those domains, and I filtered them out in my script so now I deliver them a blank page with link to my home page. Now, for example, my page title is:

Something something - mysite.com

in before, when I typed site:mysite.com in google I saw page titles as follow:

Something something - theirsite.com

but after delivering on them blank pages I can see I got my page titles back. But the problem is, when I type some keyword where I'm sure my site should be, on results page there are still those spam domains, ranking higher than me, for example

  • result 1 - theirsite.com
  • result 2 - anothertheirsite.com
  • 3 - anotheranothertheir.com
  • 4 - mysite.com
  • 5 - anotheranothertheir.com

I don't know how can I delete permamently them from search results and how can I regain my ranks?

There's more: I can add those sites to my search console (but I think I can't do anything with that)

When I go to links to your site in search console, in How your data is linked I can see that it's reading those sites as if they were mine... because some weird keywords pop up, like "phone sex taboo" and MANY MANY others.

Also, googlebot is accessing pages that don't exist on my site that I'm sure are not linked anywhere in my script, for example it's accessing: mysite.com/some-old-keyword-from-spam-sites and giving me emails and notificatinos that I should fix this error.

I think that googlebot is accessing those spam domains' old links (that were before they started scraping my site) and reading their sites as mine (as if it belongs to me).

I tried DMCA report but they say they don't see my content on theirs (because I blocked those sites), tried spam report with no results.

Like I said I lost almost all of my traffic, in one day (according to search console) my search clicks went down from 8000-12000 daily to just 100. Has anyone encountered a similiar situation before? What can I do to regain my ranks?

I can just add that those sites outranked me on most long tail keywords, and on main keywords (3 keywords) google completely removed me from search results...

Improve this question
15
  • 1
    These things happen. Oddly, this was a huge problem years ago and Google solved this problem by considering the first found content as the original. However, since the addition of the canonical tag, it appears that Google has gotten stupid again by allowing spammers to scrape sites and out rank them. This is soooooooo very wrong! Scrapers destroyed 2 of my 8 PR sites with over 10,000 unique users per day back during the bad old days. It looks as if these days have returned. Look at Fail2Ban or ModSecurity to stop the scrapers. I no longer have confidence in Google honoring DCMA like they did.
    – closetnoc
    Mar 8, 2017 at 0:56
  • 1
    There are a lot of anti-scraping techniques you can use. Search Google for stop web scrapers. Some are good and some are bad. I use a rate limiting script that runs as the page is built. If a scraper is suspected, it will replace the content portion of the page with junk. I monitored it closely for a while to make sure there were no mistakes. As well, I white-list some IP address block from known sites such as search engines. There is more to my script. For example, no image download? No content. It is that simple. However, I would suggest not rolling your own. Use a product. Much safer.
    – closetnoc
    Mar 8, 2017 at 1:05
  • 1
    One cost of running an anti-scraping software is that links to your site will disappear. These are junk links, however, Google is still considering these as valuable. Few links? Your site must be junk. So I went from over 8000 links to just a few hundred and then got pinged by Panda. Cannot win - huh? Google has really pewed the scrooch in the past couple of years. Just a warning! Cheers!!
    – closetnoc
    Mar 8, 2017 at 1:08
  • 1
    "now I deliver them a blank page with link to my home page." - can you not redirect them instead?
    – MrWhite
    Mar 8, 2017 at 8:15
  • 1
    I tried DMCA report but they say they don't see my content on theirs (because I blocked those sites), tried spam report with no results.? err? please explain.
    – Simon Hayter
    Mar 8, 2017 at 9:14

2 Answers 2

Reset to default
3

I think you have done the correct thing by serving up a blank page. It looks like Google is just being slow to update. If those scraper websites have little traffic then Google won't be visiting them often and it could take months before every page is removed.

However, I would suggest removing the 'nofollow' on the page. With nofollow the link is not counting towards your site - if you remove it, your PageRank should be boosted a little more and help to keep you above future scrapers you haven't blocked yet.

You also said the scraper sites are removing all JS code but if you're able to put other HTML on the page you could consider either a meta refresh tag:

<meta http-equiv="refresh" content="0;URL='YOURURL'" />

Or a JavaScript onload attribute:

<body onload="window.location.href='YOURURL'">

According to this meta refresh is observed by search engines so again may speed up removal of the scraper sites.

Improve this answer
6
  • So I should add this to the blank page I'm showing on those proxies or my main site? I added nofollow because link to my page as you can see is a 'goo.gl' link and I saw in some search results that instead of the my site (after redirection) the 'goo.gl' link was indexed. Don't know if that would hurt my site or not, it was redirecting to my website fine. Can't do it without goo.gl or other link changers, because as I said they remove all 'mysite.com' to 'theirsite.com'
    – buli
    Mar 9, 2017 at 21:17
  • Also, that meta refresh is the same thing I did with PHP I think, the redirection. One more thing I forgot to mention: because I can add meta tags and files on my ftp can be accessed from their's domains too, I can verify those sites as mine on search console. Can I do something with that? I tried to do the "domain change" there but it wouldn't let me, cause I need 301 redirects to my site, but like I said, can't redirect them directly to my domain, and google is checking only the first redirect not the second.
    – buli
    Mar 9, 2017 at 21:24
  • It may not work in this case, however, one primary advice is to add one or more links to your site within the content itself. Many spam software will remove these. However, I imagine that a URL shortener can possible disguise this and you may be able to bypass any link removal. Not sure. But worth a try. I like replacing the content. I replace the content with an obvious notice, however, if the OP is a clever programmer, they can simply read in their own content and scramble the words or even spin some of it. I like wasting spammers time. ;-)
    – closetnoc
    Mar 10, 2017 at 1:14
  • @monster1777 Yes I meant you should add the meta tag to the 'blank' page. Not sure about nofollow, I thought the goo.gl link used 301 redirects so it should show your URL. I don't think it's a big problem. Meta refresh is different to what you've already done - the meta tag is for the visitors to the scraper site (blank page); the PHP stuff you've done is for the scrapers visiting your original URL.
    – DisgruntledGoat
    Mar 10, 2017 at 14:34
  • Ok, so I've written some code, added a meta tag that redirects user after 3 seconds. I added "NOINDEX" robots meta tag so google won't (hopefully) index their site. I removed nofollow, so google can index the goo.gl links. I also added a goo.gl shortener code that when someone enters from some keyword they are redirected to the correct page. Don't know if I did good or wrong, hopefully it will work, their site will get deindexed...
    – buli
    Mar 11, 2017 at 16:12
0

I would also look into CloudFlare. While it won't help the situation you are currently in, there are tools on there that help prevent scraping.

Improve this answer
1
  • Yeah I got all the scraping options on, and it doesn't do a thing.
    – buli
    Mar 11, 2017 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.