In my logs, I keep noticing that
184.108.40.206, which is Verisign, is spidering through every page on my website, excluding pages of forms. Does anybody know why they are doing this?
Other people have experienced similar things: http://www.forumpostersunion.com/showthread.php?t=4163
Specifically highlighting the last post, where the bot appears to be spidering things high in SEO ranking. My personal opinion, is that someone is spidering websites and in fact spoofing their IP to hide who they are. I don't believe VeriSign rents out infrastructure or servers, so either they have been compromised to someone is putting the blame on them.
My personal recommendation, is to simply ban the bot (not using a robots.txt file since I'd almost guarantee it ignores it) by IP address (some people report it using the entire range
69.58.178.* and not to worry too much about it unless it's slowing your website down.