CAPTCHAs are not bot-proof. There are bot algorithms out there that are perfectly capable of breaking CAPTCHAs, and they're getting better and more widespread with each passing day (while CAPTCHAs have basically reached their maximum opacity). Though in my experience CAPTCHAs can still keep 98% of bots out, so it's unfortunate that your site has been particularly hard hit.
One of the things with CAPTCHA-reading algorithms is that (the last I read about them) the best algorithms only had a 50% accuracy, and the typical algorithm was more around 20%. That's not particularly high compared to the average human, but if you only need to solve the CAPTCHA once and can make as many attempts as you need, then it's plenty high to break the CAPTCHA authentication.
If you're faced with bots that can break CAPTCHAs, then you might consider putting a delay between re-issuing CAPTCHAs. E.g. if they fail the first attempt, then they need to wait 5 seconds before trying again. If they fail a second time, then they have to wait 10 seconds. If they fail a third time, then they have to wait 30 seconds. And if they fail that, then their IP is banned for 24 hours. And if a particular IP address is failing two or more times on almost every post attempt and continues to post regularly, then you can be pretty sure that it's a bot, since most people would be fed up with having to solve that many CAPTCHAs to make a forum post.
Also, you should check the strength of your CAPTCHAs, i.e.
- Never reissue the same CAPTCHA twice (even with randomized noise).
- Never use a fixed pattern answer (e.g letter, number, letter, number).
- Similarly, don't use math problem CAPTCHAs with a predictable format.*
- Make your CAPTCHA adequately long (at least 5-8 characters).
- Never use a fixed-width font. The most problematic step in the CAPTCHA-solving process for bots is segmentation. This is where bots have the lowest success rate. By using a fixed width font, you're basically getting rid of this step for them.
*One possible exception to the math problem CAPTCHA rule is if you make the CAPTCHA really long by spelling out all parts.
Additionally, you could try using word riddles (so long as they can be generated randomly) and adding accent marks to the letters in your CAPTCHAs. You'll probably also want to use other anti-bot techniques like including hidden fields to trap bots or clever JavaScript to set cookies.
Given the volume of spam you're getting, I assume your site must be very popular. So you could also enlist the help of users to help de-spam the site. If you employ a peer-moderation system in addition to a dedicated team of moderators, that should help keep spam levels to a minimum.
Though you should still consider requiring sign-up in order to post. Any serious forum poster is gonna want to sign up so they can be notified of replies. It's also more convenient to do a quick sign-up than to have to fill out a CAPTCHA every time you want to post a message. Right now you're basically catering your forum to users who are more likely to be trolls who don't want any long-term commitment to the community than serious participants.