I agree with getting the saddle etc checked.
I would also try mounting and dismounting from the off side. All horses should allow you to mount and dismount from this side in case there is ever a need for you to do it (at the side of a road or similar). I'd suggest practising on another horse if possible so you can do it smoothly.
My guess is he suddenly looses sight of you in his right eye as you move left to dismount and panics at where you have gone. If you dismount where he can see you then it might well eliminate the problem.
I would also try mounting and dismounting from the off side. All horses should allow you to mount and dismount from this side in case there is ever a need for you to do it (at the side of a road or similar). I'd suggest practising on another horse if possible so you can do it smoothly.
My guess is he suddenly looses sight of you in his right eye as you move left to dismount and panics at where you have gone. If you dismount where he can see you then it might well eliminate the problem.