The tag has no wiki summary.

learn more… | top users | synonyms

1
vote
0answers
25 views

Techniques to prevent apps hijacking my search engine?

I'm working on a site with a large music content database, and recently some app developer looking for a bit of a reputation boost has launched an app that completely piggybacks on our database: a ...
5
votes
5answers
764 views

Which tools can help limit maximum page view per IP to limit scrapers and bots?

I would like to prevent scrapers grabbing all my content except Google, Bing and other search engines. I am thinking of going with Fail2ban and limiting hits from an IP maybe at around 1000 per day. ...
1
vote
2answers
230 views

Who is Automattic and why are they visiting my non-Wordpress site so often?

Of the various people and organizations that have engaged in scraping activity, Automattic is one of the strangest. They have numerous independent IP ranges, as if they know they will be blocked so ...
1
vote
2answers
165 views

Why is Verisign spidering my website?

In my logs, I keep noticing that 69.58.178.57, which is Verisign, is spidering through every page on my website, excluding pages of forms. Does anybody know why they are doing this?
3
votes
2answers
242 views

How this site is ranked so high in SERP's with no content?

In last few months whenever i search for computer wallpapers (using various keywords), somewhere between 1st and 5th position is always site http://hdw.eweb4.com Everything on that site is scraped ...
0
votes
0answers
17 views

Is this Anti-Scraping technique viable with Crawl-Delay? [duplicate]

Possible Duplicate: How do spambots work? I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for ...
5
votes
4answers
482 views

How to protect SHTML pages from crawlers/spiders/scrapers?

I have A LOT of SHTML pages I want to protect from crawlers, spiders & scrapers. I understand the limitations of SSIs. An implementation of the following can be suggested in conjunction with any ...
1
vote
2answers
1k views

Embedding/scraping the Rotten Tomatoes “Tomatometer” score on another site

I run a local community site, and one of our services is a movies page with showtimes for local theaters, as well as links to the official site, IMDB page, Rotten Tomatoes page, etc. for each movie. ...