Need help with 200,000 + 404's

Question

The business I work for recently migrated our e-commerce site from Celerant to Magento. Despite using a robots.txt file and a 301 redirect extension, webmaster tools is reporting that our "not found" errors are increasing by the day- all from our old site. I've tried 301 redirects which don't seem to be working and then blocking the url's with the robots.txt with Disallows including wildcards but the errors continue to increase. Any advice would be greatly appreciated. Here is an example of one of our 404 url's:

http://www.bcsurf.com/category.cfm/brand/paris

This is how we tried to block it in our robots.txt. Is this correct?

Originally we used- Disallow: /category.cfm

A few days ago we decided to try- Disallow: /*category

Any help would be appreciated. Thank you.

Answer 1

Your original syntax for robots.txt was the correct syntax: Disallow: /category.cfm Webmaster Tools reports things that Googlebot may have crawled some time ago as they get processed, so your 404 errors could grow even after you are not allowing Googlebot to recrawl the urls.

301 redirects really are the correct way to handle these types of things. In the one example you gave for the paris brand page, your search engine seems to do a decent job of handling that: http://boardshop.bcsurf.com/search?w=paris I'd recommend 301 redirecting that brand page to your search engine.