The tag has no wiki summary.

learn more… | top users | synonyms

5
votes
3answers
653 views

Someone has cloned my WordPress blog, how do I prevent it from hurting SEO?

My WordPress blog is completely cloned. That clone site is updating in real time with my blog. I am surprised that someone can actually do that. What should I do to stop harmful impact in my search ...
5
votes
5answers
1k views

Which tools can help limit maximum page view per IP to limit scrapers and bots?

I would like to prevent scrapers grabbing all my content except Google, Bing and other search engines. I am thinking of going with Fail2ban and limiting hits from an IP maybe at around 1000 per day. ...
5
votes
4answers
507 views

How to protect SHTML pages from crawlers/spiders/scrapers?

I have A LOT of SHTML pages I want to protect from crawlers, spiders & scrapers. I understand the limitations of SSIs. An implementation of the following can be suggested in conjunction with any ...
4
votes
2answers
141 views

Scraped content from my website posted on Blogspot is outranking my site and causing Google penalty

I noticed that starting from October the organic traffic is very low on my website. I searched on Google for some text phrases from my website and I noticed that there is a Blogspot website, a blog ...
3
votes
2answers
282 views

How this site is ranked so high in SERP's with no content?

In last few months whenever i search for computer wallpapers (using various keywords), somewhere between 1st and 5th position is always site http://hdw.eweb4.com Everything on that site is scraped ...
3
votes
1answer
46 views

Techniques to prevent apps hijacking my search engine?

I'm working on a site with a large music content database, and recently some app developer looking for a bit of a reputation boost has launched an app that completely piggybacks on our database: a ...
1
vote
2answers
41 views

Can I limit content scrapers by counting the number of hits from an IP?

I would like to ban aggressive scrapers that access x number of pages in an hour, say 1000. I plan to implement such a check via fail2ban and count the hits from the same ip via access logs. Should I ...
1
vote
2answers
274 views

Who is Automattic and why are they visiting my non-Wordpress site so often?

Of the various people and organizations that have engaged in scraping activity, Automattic is one of the strangest. They have numerous independent IP ranges, as if they know they will be blocked so ...
1
vote
2answers
276 views

Why is Verisign spidering my website?

In my logs, I keep noticing that 69.58.178.57, which is Verisign, is spidering through every page on my website, excluding pages of forms. Does anybody know why they are doing this?
1
vote
2answers
1k views

Embedding/scraping the Rotten Tomatoes “Tomatometer” score on another site

I run a local community site, and one of our services is a movies page with showtimes for local theaters, as well as links to the official site, IMDB page, Rotten Tomatoes page, etc. for each movie. ...
0
votes
0answers
8 views

Considering what server/cloud service for a responsive website [duplicate]

The website will be internationally used by it's users. The website will have a database containing Images and text. The website is for price comparison between products. Data will be Scraped from 60 ...
0
votes
0answers
18 views

Is this Anti-Scraping technique viable with Crawl-Delay? [duplicate]

Possible Duplicate: How do spambots work? I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for ...