If it counts for anything, I'll insert my $.02...
My friend's got a horse that's about 9 years old, stands around 15.1hh, and is a little SNOT! He strikes out in front (shod, as well), can't be tied high to a wall/post, or he goes NUTS and pulls back until he either breaks something, or gets loose. It takes 2 people to saddle him, because he turns circles (can't be tied for saddling, of course), is VERY high strung and to me, dangerous.
My horse is 3 1/2, stands 17+hh, can be tied high to a wall/post, will move around a little bit during grooming, but nothing quick. Can be saddled without worry, bridling is done without her being tied to anything (just loop the lead around her neck for the "caught" feel), walks away from the mounting block (we're working on this problem).
Personally, I'd MUCH rather see someone inexperienced around my much larger, younger mare than the smaller, friskier gelding. She's much safer, saner and forgiving than he is. No, it's not ALWAYS that way, however, there ARE times when gravity shouldn't have as much to do with a horse as personality/temprement.
And as far as your looking silly on a big horse? I wouldn't worry over it. How you FEEL on and around the horse is SO much more important!!
***oh, and I ride western (I'm about 5'6") and have to heave a western saddle that weighs between 35 and 40 pounds up on her every time I ride. It's not easy, but it IS great exercise!