I nearly converted to brown, but I honestly prefer black. My pony is a bright red bay with black points and black does seem to suit him. I think that most judges would look at the horse first, and the tack and riders clothes second. I would only think it would matter if there was very little to choose between two animals.
Vaguely funny story - I was at a show and an saying that I had finally got the dosh for the saddle of my dreams an Albion selecta (second hand). An acquitance then started to bang on about having spent £1,200 (what my pony cost) on a brown saddle as she needed it as she has a reg. cob (unlike mine) and they show county dont you know. She came seventh and we came third which sort of summed it all up really!