i have no problem with my horses walking politely WITH me, either.
youth needs to touch when afraid or confused. instinctual. he gets scared and forgets his space. he's looking to you to save him. do your best to know what's around you when he's in hand, know the things that will unsettle him, be prepared. put him back where he belongs but not in a disciplinary way, just business-like. "this goes here", so to speak.
spend some time sacking him out, and desensitising him to as much as you can. this will help minimizes the eruptions, tough him up a bit. teach him self-confidence with standing exercises, ground tying, working him increasingly away from you. be patient, clear and consistant.