I don't find mine restictive - had a job getting one too fit me, but the Airowears come in so many different fittings there was one that was okay, even if the velcro fastenings round the waist are too long!
However, I only wear mine for competing cross country (not XC training on my own), or jumping a horse that is a stopper, as it's just too hot otherwise, and I'd feel odd hacking out or doing flatwork in one. I know that's a completely rubbish reason, but hey.