You did the right thing. I've had this happen to me too. You have to wonder who would want to be with someone who obviously does not want to be with them.
That's true.
And if he liked me so much, he should have acted like it more.
You did the right thing but that doesn't mean it was easy. I know far too many people stay in a bad situation because they don't want to hurt the other person's feelings.