When you're talking with him in a social situation you're talking as friends. I think he might be afraid of losing that. Losing you as a friend if he did start dating you.
As for what you can do about it, I don't know. You could even ask him out on a date. Take the initiative. But I like 90802's suggestions too. Definately let him know what you want.