Tag Info

Hot answers tagged

8

They're simply loading your site via a server-side script. All you need to do is block their server's IP address via .htaccess. Simply open up your server's access logs, open the cloned page on their site, then view your log for the new entry and you'll have their IP address. It also wouldn't hurt to submit a DMCA request to Google as well but this will not ...


5

(In addition to @John's answer.) Is there any way to tell Google not to index that site? Rather curious that whilst they appear to have cloned everything (including your XML sitemaps*1), they have not cloned your robots.txt file. In fact, the robots.txt on that site actively blocks crawling of everything! So there would not seem to be anything to do in ...


5

There are many ways this can be done within Apache using Modules, or alternatively you can setup IP tables to do the job though personally I just use the modules. mod_security I've personally used this and it does the job well, a good article about limiting requests can be found here. mod_evasive Detection is performed by creating an internal dynamic ...


4

It's not normal that your site is outranking you, but can happen. Google has its own tool to report Scrapped content, you can check it here -> https://docs.google.com/forms/d/1Pw1KVOVRyr4a7ezj_6SHghnX1Y6bp1SOVmy60QjkF0Y/viewform I also advise you to make a DMCA complaint, as addresses copyright issues and can act faster. I had that experience in the past ...


3

Enter that website's url into www.opensiteexplorer.org (or any other backlink checker) and you'll see it has lots of backlinks from a variety of domains. You can further use other toolboxes (sistrix, searchmetrics, xovi, seolytics...) to analyze the value of the site over a period of time to see if there was a sudden spike of links or if the link building ...


3

If the site produces backlinks to you it is important to use the Google Disavow tool otherwise the algorithm will be working against you, regardless. https://www.google.com/webmasters/tools/disavow-links-main create a .txt file and add: domain:thedamnsitethatcloned.com then upload it to Google via Webmaster Tools. Here are exactly the steps that I ...


1

Alas if your data is worth getting it will be gotten. When we decided to handle IP addresses, the scrapers went horizontal on us and came back with thousands of IP addresses, each accessing us well under the limit we set. I am myself hunting for the newest hints on how to stop robots that do not respect any protection.


1

Yes. You can. I do not know the tool you are using, but here are some clues. Not a search engine (duh!). High number of requests over a longer period of time- more than a user. Does not request an image, CSS, or JS (some do, most don't). Changes agent name and OS over a longer period of time. Comes from a web-hosting IP address block/range. Check for ...


1

Your token idea would work. You could change it periodically to make it harder. You could also change the parameters on your search form. Start using s= instead of q= for the search term. That would force this developer to keep up with those changes. You could also implement Captcha and not show search results unless the user proves they are a human. ...


1

make a spider trap: make a page like /spider-trap/somepage.html block the page on robots.txt: Disallow: /spider-trap/ place a link to this page but hide it to human eyes block the IP for ANYTHING reach this page show a human readable hints and an unlock IP captcha something on this page.


1

Why bother scraping when you can use their official syndication methods? This is much less likely to piss them off, and should be fairly easy for any developer to parse and integrate into your site. This way a change to their HTML doesn't break your site. This is also much less likely to piss off the RT guys, meaning you're less likely to get banned.


1

A CDN service could sit in front of your site and filter out known crawlers; they also filter out spammers and since they cache your images at locations around the world your site would be faster. I have been using CloudFlare for about a month on a site for a client and they have seen a decrease in bandwidth use, and an increase in traffic. CloudFlare also ...


1

Automattic Analytics Crawler At Automattic we’re passionate about creating the greatest publishing platform the world has ever seen. We think we’ve done a pretty good job so far, but we’re not resting on our laurels. We’re constantly striving to make WordPress better. One of the more effective ways for us to do that is to listen to our users. The ...


1

It could be related to the daily malware scan[CNet, PDF @Verisign] they run automatically as a service to their ssl certificate customers. You could always opt out of the service temporarily in order to see if the crawls from that IP cease.


1

Following might be the reasons for this site showing higher in search results: 1) This site may be getting lot of back-links from other sites. 2) As per the recent Google Panda update, if one site is linking to another sites, which can be useful for the viewers, then Google happens to give edge to that very site ( In this case it is there). 3) Last but ...



Only top voted, non community-wiki answers of a minimum length are eligible