You definitely must set limits for yourself and living 10-12 hours away you obviously can't be driving home enough to take care of your parents' garden. I also like living far enough from my parents to not feel obligated to visit too often and I think it makes our relationship better. But it is not a good idea to get into a toxic exchange with someone over this issue. It sounds like he attacked you and you felt very defensive and so lashed out with a counter-attack; this type of exchange is unlikely to have any positive results.
You admit that you got all worked up over this exchange and it made you angry and unproductive, so it sounds like you need to find a better way to say NO in the future. Erin had a
blog entry recently where she shares her favorite NO response, “I’m sorry, I’m overcommitted right now. I’m unable to help you.” Wayne Dyer has shared his favorite NO response, "Thank you for considering me, but I am not able to take that on at this time."
I would agree with Clarity that if possible it would be nice to hire someone to help. I hired someone from craigslist recently for only $10/hour and he worked so fast that he did a job that it would have taken me about 20 hours in only 6 hours. What a bargain!