What is your body language saying to your horse whenyou approach him? Are you going straight up to him like a predator then stretching out your claws (hands) to attack him?
Try approaching him slowly from the side. If he looks worried like he is goint o walk off then stop for a moment, then approach again. Keep your shoulders lowered and slightly turned away from him. When you get to his shoulder, slowly move your arm to his shoulder, keeping your fingers folded up then gently stroke his shoulder. If he is happy with that then try to run your hand further up his neck. If he looks worried, go backa step, then try it again. It will take time and patience.