Letting go is right.
You may think you have to do something from a scarcity mindset.
Basically, you're probably thinking "Well, I might not get the chance to lucid dream in a very long time, so I better make the best of this. What should I do? What should I do WHAT!!??" It might be subconscious or conscious, but whatever it is, this is probably part of the thought process that leaves you scrambling to do something in the lucid dream.
So cut that stuff out

!
You may also try figuring out what you'll do before sleeping, perhaps something that excites you.