What a load of responsibility you must feel and yes you are wise to set boundaries.
Really the problems are theirs and they should be made aware and responsbible for caring for things. Do you think things in the garden had gotten out of control?
Is there a way for them to pay a gardener?
I think I'm hearing a sad little girl that didn't get affirmation. Chances are you were of a personality different than them. Maybe a gene passed through another generation. One of a more sensitive nature for whom approval and affirmation were very important. When coming to terms with this sometimes one merely needs to accept that their parents brought them up the way they knew and they knew no better. If their personality isn't as sensitive as yours they would also have no clue the level of attention you were lacking.
Also if you are a more sensitive compliant personality you may have seen the world as much harsher than it was or than it would be for a strong willed child. So as a grown up it is important to balance all of this and come to a place of forgiveness and acceptance.
Just do your best. You don't need to do it all.
Throw out the letter and every time it creeps back into your mind change the channel.