In my logs, I keep noticing that
184.108.40.206, which is Verisign, is spidering through every page on my website, excluding pages of forms. Does anybody know why they are doing this?
It could be related to the daily malware scan[CNet, PDF @Verisign] they run automatically as a service to their ssl certificate customers. You could always opt out of the service temporarily in order to see if the crawls from that IP cease.
Other people have experienced similar things: http://www.forumpostersunion.com/showthread.php?t=4163
Specifically highlighting the last post, where the bot appears to be spidering things high in SEO ranking. My personal opinion, is that someone is spidering websites and in fact spoofing their IP to hide who they are. I don't believe VeriSign rents out infrastructure or servers, so either they have been compromised to someone is putting the blame on them.
My personal recommendation, is to simply ban the bot (not using a robots.txt file since I'd almost guarantee it ignores it) by IP address (some people report it using the entire range