The business I work for recently migrated our e-commerce site from Celerant to Magento. Despite using a robots.txt file and a 301 redirect extension, webmaster tools is reporting that our "not found" errors are increasing by the day- actually 200,000+, all from our old site.

I've tried 301 redirects which don't seem to be working and then blocking the URLs with the robots.txt with Disallows including wildcards but the errors continue to increase. Any advice would be greatly appreciated. Here is an example of one of our 404 URLs.

This is how we tried to block it in our robots.txt. Is this correct?

Originally we used:

Disallow: /category.cfm

A few days ago we decided to try:

Disallow: /*category
  • What web server are you using?
    – Tim Post
    Commented Feb 22, 2013 at 0:42

1 Answer 1


Your original syntax for robots.txt was the correct syntax: Disallow: /category.cfm Webmaster Tools reports things that Googlebot may have crawled some time ago as they get processed, so your 404 errors could grow even after you are not allowing Googlebot to recrawl the urls.

301 redirects really are the correct way to handle these types of things. In the one example you gave for the paris brand page, your search engine seems to do a decent job of handling that: http://boardshop.bcsurf.com/search?w=paris I'd recommend 301 redirecting that brand page to your search engine.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.