Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Of the various people and organizations that have engaged in scraping activity, Automattic is one of the strangest. They have numerous independent IP ranges, as if they know they will be blocked so they want to make it hard. But who are they? Why are they scraping my site?

Update By scraping I mean excessive unwarranted visits to my non-WordPress site. Like visiting the same page 10 times in under a minute. Or visiting every day. I've had to ban their many IP ranges, which are ranges located in many geographical regions.

share|improve this question
    
Automattic is most famous for running WordPress.com (Wiki Page | Site). What exactly do you mean when you say they're scraping your site? –  BigBagel Mar 27 '13 at 2:36
    
Yep, they're big. The only companies with a larger web reach are Google and Facebook. I also want to know what you mean by scraping as they're a very reputable company. –  Jeremiah Prummer Mar 27 '13 at 3:55
    
That is not scraping. Also, have you dug a user-agent or anything out of your logs? It might give some indication what exactly is behind the visits. (Bybe's answer is just speculative right now, though seems likely.) –  Su' Mar 28 '13 at 3:26
    
Maybe the keep checking to see if you will start using Wordpress? ;) –  Itai Mar 28 '13 at 13:26
1  
Well anyway, I'm redirecting them to a Femen page with pictures of attractive, topless feminists. –  Funnelcake Mar 29 '13 at 2:45
add comment

2 Answers

Automattic Analytics Crawler

At Automattic we’re passionate about creating the greatest publishing platform the world has ever seen. We think we’ve done a pretty good job so far, but we’re not resting on our laurels. We’re constantly striving to make WordPress better. One of the more effective ways for us to do that is to listen to our users. The Automattic Analytics Crawler is a utility we use to discover how people use WordPress. Please let us know if you have any questions about the crawler.

Source: http://en.wordpress.com/crawler/

share|improve this answer
    
That still doesn't saw what they do with the info and is hardly an unbiased description of what they do ;) –  Itai Mar 27 '13 at 13:14
    
'The Automattic Analytics Crawler is a utility we use to discover how people use WordPress' - means that the crawl your website and determine how your using WordPress of example, niche type example... very its board I don't think your find a list it does this, it records this as much crawlers don't disclose this information. –  bybe Mar 27 '13 at 13:19
1  
@bybe: Here's an important fact: I don't use WordPress. I have linked to my site just once or twice when I've posted comments in WordPress blogs, but I don't think that warrants the excessive visits they make to my site. –  Funnelcake Mar 28 '13 at 2:47
add comment

As far as I know, Automattic are generally regarded as a reputable company, so perhaps you should contact them and ask for an explanation. Maybe there is a bug in their scraper.

However, given you said the IPs come from a variety of locations (I assume you mean many different countries) it's very possible that this is not Automattic making these requests at all and in fact a spambot etc.

In which case you should continue to block the IPs making bad requests. I would advise against blocking too general and IP range though - you could end up blocking real visitors.

share|improve this answer
    
Interesting hypothesis however this would require nefarious persons to be entering fake registration information, because I've been learning it's Automattic based on whois lookups. –  Funnelcake Mar 29 '13 at 2:44
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.