Maybe you should try doing a "pros" and "cons" list. And be honest about it and see what you come up with.
I think your gut is telling you what you really need to do, but your heart is playing mind games with you. He's not hurting you, you are hurting yourself sweetheart.
