Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

On one website I have, looking at the logs, I find lots and lots of "Sosospider" hits. Is this the same for everyone, or just me? Now, I have never once been sent any traffic from anything which looks like it might be anything to do with Sosospider, so can anyone explain what this thing does? Does it cache the pages and send cached pages to users or something?

share|improve this question
    
The problem is that they send the Sosospider to my site 40 to 50 times/hour. I dont mind they crawl my site but wtf: up to 50 times/hour!!!! Soon I will just drop the incomming IP-packet in the firewall. –  Peter Feb 1 '11 at 23:50

2 Answers 2

It looks like it is just a search engine crawler for a website in China. The lack of traffic may have to do with the fact that your site is in English (assumption) and probably isn't what its users are looking for. I would think the crawler wouldn't crawl you since your site is not in Chinese (assumption) but maybe their system isn't that sophisticated or doesn't care.

share|improve this answer
    
Yes, the site is in English. I also get some hits from Baiduspider but I think I have rarely had an actual hit from them; I have this suspicion that they are caching my pages and serving the cached pages to their users. –  delete Oct 16 '10 at 13:22
    
Just because you aren't getting any traffic from them doesn't mean they're stealing your content. They're likely just trying to expand their international index. Many Chinese people can read English, just as many Americans read foreign languages. So it makes sense for search engines to index foreign sites. Even if no one is searching for these pages yet, they could in the future. And spidering more pages helps them build better link graphs for ranking results, even if not all pages will be useful results in and of themselves. –  Lèse majesté Jan 28 '11 at 12:03

Sosospider is a Chinese Search Engine. Whether you think it is worthy of crawling your site depends on your sites contents. My site is an online retail shop which does not ship to China, so we do not allow them to crawl it. However, if you decide to ban them and implement a robots.txt file, keep an eye on your logs because many have said they ignore the robots.txt (regardless of what their website states).

share|improve this answer

protected by John Conde Jan 18 '12 at 23:30

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?