Wish I hadnt bothered now. YES I SHOULD BE WEARING HI VIZ. I am actually starting to borrow some, so no comments needed there. My annoyance is with drivers lecturing me about it - its none of their business about my personal choice. Maybe if they almost hit me due to no other factor than I wasnt wearing hi viz they may be able to mention it. But 9 times out 10 a close shave will be due to them speeding/being inattentive or similar.
Do you shout/lecture or get annoyed with drivers because they come too close to you, or maybe don't slow down enough when they come past you? They could claim it was none of your business how fast they were going, or how close they went to you. Which of course we know is silly, because it is your business, because it's your horse they're going to hit. It's the same for hi-viz, it is their business, because it's their car that's going to be damaged, and them that's going to be blamed if they hit your horse because they can't see it!