Yossarian, as usual, your articulate mind spelled it all out in simple terms. Thanks for the good advice.
Freshwala - this is actually the first time in 6 years I have been to my brother's, and if not for a chance meeting where we were both at my mother's one day, it would have been 6 years since I last saw him - even so, that meeting was a couple years ago, and it was only a day. I have no problem with staying away from him. My father is even far worse and I have no contact with him anymore at all. The problem is that I'm at a bad point in my life and I'm in sort of a helpless position. If I had the presence of mind in the moment, I could always react like Yossarian wrote, which enables standing up for myself. I guess I won't be here much longer though. I have probably benefitted more than anyone from my brother having kids though, because he expends the vast majority of his "control" resources on controlling them
and he has lightened up on me exceedingly since he had them. Doesn't make me want to be around him any more though, I have enough problems just being alone.