What's more empowering? Being able to control your environment, or being able to control how you interact with it?
By holding a mindset that stresses you out whenever your environment is cluttered, or the people around you are ornery, aren't you then giving your power away to your environment?
What exactly are the best parameters for personal growth? Is it existing in an environment where you can control everything? Or learning to leverage the variables you have unconditional control over by exposing yourself to uncontrollable environments? Would you grow more from spending the rest of your life on the Holodeck—where EVERYTHING is under your control? Or by vagabonding and going wherever the wind blows you?
Just as no one can make you feel inferior without your consent, nothing can stress you out without you focusing on it in a way that creates resistance. If you can make peace with any environment, you can find peace in any environment. If you can make peace with any person (no matter their attitude), then you are free to focus on any aspect of any person in your environment and create interactions accordingly, because you're making what resonates with you most the topic of discussion, not what annoys you most.
Growth is everywhere. The better you can get at controlling your focus, the more freedom you'll have from the need to control your environment. You'll be able to enjoy yourself in the finest of restaurants seated next to the most conscious of people, and in the dingiest of prisons seated next to the most bitter of prisoners. Because the Oneness within us all means that the good within everything we love exists in every particle of the world around us. And it is there for us the conscious to find it, embrace it, and reap its benefits.