I was 24 when I lost my virginity. And I was scared as hell when it happened! My spouse was actually a virgin as well and we waited until we were married. For the majority of my life i was a christian, so that was my main reason in waiting. However, at the time I got married, (and about a year and a half beforehand) I was not a christian, but we felt that since we had waited so long, we would wait until marriage. And now, it's like, "whatever" when it comes to having sex.
People build it up to be this HUGE thing, and really, it is only huge because you do it with someone you love or care about. Now, some people have sex without love, and that's fine because it's their right to do so. But I have a feeling that you wantto feel that emotional connection with a person, and that's wonderful.
Something you should consider though, when thinking of having sex with this person you love who is coming in for a day--if you really see the relationship going nowhere, then is it worth it to have sex with him one time because you want to show him you love him, or would you rather wait to experience it with someone you can see daily and have sex with on more than one occasion

? If you know you will have anxiety over him leaving without having sex, then I don't know if it would be worth it to have sex and be anxious about that as well.
You know your heart and your heart seems to be telling you that you want to do it, but you want to wait for the right person. I'm not saying to wait until your married, but definately make sure that you wait until it feels right and you have the right man to encourage you through it.
BUT, ulitmately, it's up to you and the person you're going to have sex with. And honestly, it's not all that bad. It's so scary at first because you've never done it before, and it seems that the older you get the more anxious you become about having it. My spouse and I were like awkward 14 year olds, but it didn't matter because we loved eachother and that's why we were doing it.
You'll know when the time is right, you'll just feel it...