First off, just as Marinik was saying, we all play our games.
And though I wasn't there, I get the impression you were both doing something similar from your post...
You paid attention, and then played aloof. He seemed to be doing the same thing.
Did you ever come out and state your intent/interest clearly?
I understand this isn't what people typically do... but if you sincerely want to avoid 'games' that is what you should do.
In most contexts, 'games' imply fun.
I find it interesting that people usually don't complain about games unless the game didn't turn out the way they would like...
But that's just my impression...
keep smiling,
Ben