I faced exactly the same struggle for decades. I finally figured it out.
The basic theory is that negative emotions are created by having desires that go unfulfilled. Attempting to overcome those negative emotions and the pain that they cause will never end until you either fulfill your desire, or choose to abandon it, because it is not practically possible of fulfill.
We have many desires that are impossible to fulfill at the time of wanting them and this generates negative emotion and the struggle you talk of. When you are aware of your desires and manage them effectively then the struggle disappears.
On my blog there is a free ebook that you can get called 'Understand How to Operate Your Brain Perfectly' the sub title is "Understanding Internal Mental Struggle and Negative Emotions and How to Counteract Them"
You get it by subscribing to my newsletter (you can always unsubscribe afterwards). A lot people have found it
very useful.
You can get it at
www.Nick Pagan.com