Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 4 down vote favorite

On one website I have, looking at the logs, I find lots and lots of "Sosospider" hits. Is this the same for everyone, or just me? Now, I have never once been sent any traffic from anything which looks like it might be anything to do with Sosospider, so can anyone explain what this thing does? Does it cache the pages and send cached pages to users or something?

share|improve this question
The problem is that they send the Sosospider to my site 40 to 50 times/hour. I dont mind they crawl my site but wtf: up to 50 times/hour!!!! Soon I will just drop the incomming IP-packet in the firewall. – Peter Feb 1 '11 at 23:50

2 Answers

up vote 5 down vote

It looks like it is just a search engine crawler for a website in China. The lack of traffic may have to do with the fact that your site is in English (assumption) and probably isn't what its users are looking for. I would think the crawler wouldn't crawl you since your site is not in Chinese (assumption) but maybe their system isn't that sophisticated or doesn't care.

share|improve this answer
Yes, the site is in English. I also get some hits from Baiduspider but I think I have rarely had an actual hit from them; I have this suspicion that they are caching my pages and serving the cached pages to their users. – delete Oct 16 '10 at 13:22
Just because you aren't getting any traffic from them doesn't mean they're stealing your content. They're likely just trying to expand their international index. Many Chinese people can read English, just as many Americans read foreign languages. So it makes sense for search engines to index foreign sites. Even if no one is searching for these pages yet, they could in the future. And spidering more pages helps them build better link graphs for ranking results, even if not all pages will be useful results in and of themselves. – Lèse majesté Jan 28 '11 at 12:03
up vote 1 down vote

Sosospider is a Chinese Search Engine. Whether you think it is worthy of crawling your site depends on your sites contents. My site is an online retail shop which does not ship to China, so we do not allow them to crawl it. However, if you decide to ban them and implement a robots.txt file, keep an eye on your logs because many have said they ignore the robots.txt (regardless of what their website states).

share|improve this answer

protected by John Conde♦ Jan 18 '12 at 23:30

This question is protected to prevent "thanks!", "me too!", or spam answers by new users. To answer it, you must have earned at least 10 reputation on this site.