I read one author who claims that every partner we sleep with, becomes emotionally connected to you. You will experience their ups and downs as you go through life. No idea whether it's true or not, but it sounded like an interesting hypothesis.
The other idea was presented by Muktananda: if you want to achieve Samathi you will need every available drop of energy in your body, i.e. no sex.