I don't think that you've done anything wrong. It seems to me that his perspective is this: Someone he didn't know very well invited him to a party which he wanted to attend but didn't feel comfortable going to alone. He thought that you would eventually give in to his repeated texts and go with him. When he realised that you weren't going to reply, he had a choice: stay home or go to the party. However, deciding to go and not feeling comfortable around other people, he probably hoped that you would go over to him first and felt rejected that you didn't. I think that speaking to the host first was basic etiquette but I can see how this played in your friend's mind. He felt uncomfortable, alone and rejected. The situation got worse in his mind, so he decided to leave.
I actually think that if he found that experience negative enough, he may just decide not to get in touch with you. It really depends on how many friends he has. In your position, I would probably have sought him out once I got to the party but his reaction was over the top.
It sounds to me like this isn't a one-off incident and you've hinted that there's the possibility that he is looking for more from you than you're able to give. I think that you need to ask yourself this when deciding whether or not to let go of the relationship: is there anything positive that you would miss? Sure, he might be nice sometimes, but is it something that you would actually miss?
Like you, I am in the situation of letting a friendship go because tbh, I won't miss him much.