I think it is entirely down to what a particular horse or pony is comfortable with rather than set rules. My 14-ish stone partner rides my 14.2 sec D from time to time and she shows no sign of being uncomfortable, will quite happily canter, jump etc. She is an opinionated sort of mare and would not tolerate a too-heavy rider - when I first got her she came with a badly fitting saddle and she wouldn't let me mount until I got it sorted.
However, my partner once had some lessons at a local riding school and one week his regular horse wasn't available so they brought out a cobby type about the same size as mine and it was not at all happy, laid it's ears back and kicked out when he tried to walk it forward - after that they let him ride mine instead and never said anything about him being too heavy so he must have looked ok