That will absolutely have a huge effect on your rankings. By changing your URLs you are essentially deleting your old pages and creating news ones. That means any rankings and links you have to the old URLs will be thrown away.
If you do this you must do a 301 redirect from the old URLs to the new URLs. That way all links and other good stuff associated with the old URLs will be passed on to the new URLs. A small amount of it is lost but it is far better then losing everything.
The code below should work for you (you will need to make the necessary changes for your specific needs). But I am not a .htacess guru so someone may need to correct this for me.
RedirectMatch 301 /categories/(.*)/(.*)/(.*) http://www.example.com/category/$1/$2/$3