Don't forget, horrible thought, but if you fall off out hacking or in some woods, field etc, it will be much easier for someone to spot you and find you. Same for a loose horse. I always hack out with a tabard and hat band on, though I should use my horse leg bands too.
Deffo use hiviz in poor light or bad weather, mind you, but in normal daylight the difference it makes is minimal.