What if you want to leave...and he doesn't want to let you go?

Say the same thing right back to him, to give it time and that he will learn to accept you just want to be friends