I hate to break it to you, but you simply can't completely shield your kids from bad influences, whether its their dad or otherwise. Worry not about the example he sets, but worry about the example you set!
Kids are perfectly capable of seeing right from wrong.
Also, you mentioned back there that you put your own feelings aside so as to not hurt others. That's a bad policy! If you keep hurting yourself like that, you're just limiting your own ability to help those around you. Not to mention yourself...
Hang in there!