Tag Info

Hot answers tagged

5

There are many ways this can be done within Apache using Modules, or alternatively you can setup IP tables to do the job though personally I just use the modules. mod_security I've personally used this and it does the job well, a good article about limiting requests can be found here. mod_evasive Detection is performed by creating an internal dynamic ...


3

Enter that website's url into www.opensiteexplorer.org (or any other backlink checker) and you'll see it has lots of backlinks from a variety of domains. You can further use other toolboxes (sistrix, searchmetrics, xovi, seolytics...) to analyze the value of the site over a period of time to see if there was a sudden spike of links or if the link building ...


1

make a spider trap: make a page like /spider-trap/somepage.html block the page on robots.txt: Disallow: /spider-trap/ place a link to this page but hide it to human eyes block the IP for ANYTHING reach this page show a human readable hints and an unlock IP captcha something on this page.


1

Why bother scraping when you can use their official syndication methods? This is much less likely to piss them off, and should be fairly easy for any developer to parse and integrate into your site. This way a change to their HTML doesn't break your site. This is also much less likely to piss off the RT guys, meaning you're less likely to get banned.


1

A CDN service could sit in front of your site and filter out known crawlers; they also filter out spammers and since they cache your images at locations around the world your site would be faster. I have been using CloudFlare for about a month on a site for a client and they have seen a decrease in bandwidth use, and an increase in traffic. CloudFlare also ...


1

Automattic Analytics Crawler At Automattic we’re passionate about creating the greatest publishing platform the world has ever seen. We think we’ve done a pretty good job so far, but we’re not resting on our laurels. We’re constantly striving to make WordPress better. One of the more effective ways for us to do that is to listen to our users. The ...


1

It could be related to the daily malware scan[CNet, PDF @Verisign] they run automatically as a service to their ssl certificate customers. You could always opt out of the service temporarily in order to see if the crawls from that IP cease.


1

Following might be the reasons for this site showing higher in search results: 1) This site may be getting lot of back-links from other sites. 2) As per the recent Google Panda update, if one site is linking to another sites, which can be useful for the viewers, then Google happens to give edge to that very site ( In this case it is there). 3) Last but ...



Only top voted, non community-wiki answers of a minimum length are eligible