In my logs, I keep noticing that, which is Verisign, is spidering through every page on my website, excluding pages of forms. Does anybody know why they are doing this?

  • 1
    Why is this question tagged with SSL/SSL-certificate? Jan 6 '13 at 3:04
  • Probably need a bit of log detail on this one to see what's really going on. Jan 6 '13 at 4:01
  • If you're not using their services(see response from @mahnsc), it's probably just going to come down to "because they want to" and you'll have to ask if you really want to know. Anyone can run a spider. Years ago, a personal site I ran was regularly being hit by the IAEA for reasons I couldn't even begin to speculate about.
    – Su'
    Jan 10 '13 at 14:36

It could be related to the daily malware scan[CNet, PDF @Verisign] they run automatically as a service to their ssl certificate customers. You could always opt out of the service temporarily in order to see if the crawls from that IP cease.


Other people have experienced similar things: http://www.forumpostersunion.com/showthread.php?t=4163

Specifically highlighting the last post, where the bot appears to be spidering things high in SEO ranking. My personal opinion, is that someone is spidering websites and in fact spoofing their IP to hide who they are. I don't believe VeriSign rents out infrastructure or servers, so either they have been compromised to someone is putting the blame on them.

My personal recommendation, is to simply ban the bot (not using a robots.txt file since I'd almost guarantee it ignores it) by IP address (some people report it using the entire range 69.58.178.* and not to worry too much about it unless it's slowing your website down.

  • 1
    Unless the attacker is on the same subnet as you and sniffing all IP traffic, IP spoofing only works one way. You can send spoofed packets, but then the responses won't come back to you. As such, it's pretty difficult to even open a TCP connection while spoofing your IP on the internet, much less try to spider a website (which requires actually reading the responses from the web server). Jan 6 '13 at 3:03
  • I understand it works only one direction. It is possible to send GET requests via an invalid IP. But we don't know where it finds the pages to spider. If the pages trend towards popularity before being crawled, perhaps the bot is cross-referencing with search engines.
    – ionFish
    Jan 6 '13 at 3:07
  • 1
    To send a GET request, you'd need to open a TCP connection, which requires a SYN/SYN-ACK/ACK handshake with correct sequence numbers, which is pretty close to impossible for the attacker to perform without seeing the random sequence number the sever chose for the SYN-ACK. And if you can't actually capture the server response to the GET request, how are you really "spidering" the site? Jan 6 '13 at 3:14
  • IP Spoofing is highly overrated and best used for DDOS backscatter and joe jobbing where you want the replies to be directed to the spoofed IP. It doesn't work the way you think it works. Jan 6 '13 at 3:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.