I had the opposite problem to most lol ...
I was with my 'first' for over 20 years ... and I was embarassed to admit to the next guy that I had only ever been with one man.
Thought it might freak him out a little.
So I guess you can worry about it either way.