I personnally don't think this would be a reason for worry unless you have other symptoms.
The reason I'm confident about it is that I've experienced the same kind of situation frequently when I was younger. What I found out was that the more I stressed about it happening, the more it happened.
As you mentionned, when you try to calm down and slow your breathing, you get out of it. So the real solution is for you to rationalize the situation and accept that you may sometimes experience this paralysis but keep in mind it will go away on its own and it's not dangerous.