It definitely should NOT take ages to do, and you do not keep on & on until they want to come into you!!!
If your horse/pony doesn't seem to want to join up with you, then do not just keep going on at them, look again at what they are doing & what you are doing - either you are giving them the wrong body language, or you are missing the signs they are giving!
I have never done a join up which took longer than ten minutes - and that was a seriously fiesty one who needed to gallop & buck first!