You're right, as long as you can ensure that you're using UTF-8 through and through, then you shouldn't need to escape anything but the XML entities that you mentioned (<, >, &).
I think the reason you see people escaping other characters is because they've become a little "shell-shocked" by UTF-8 being converted into another encoding and having all of their beautiful characters transformed into tall rectangles, or diamonds with a question mark in them, which looks about as unprofessional as you can get.
That only has to happen once or twice in a production environment to make you start reflexively changing everything to HTML entities.
Combine that with the fact that text editors, scripting languages, and database engines have the capability of changing the text encoding on you... I can't say I blame them too much.
But, long story short - if you can guarantee that you're going to have nothing but UTF-8 from source to served pages, there isn't any technical reason to escape anything.