I did this 3-4 times in my life. Some people thought I was nutso but I think most of them were a bit jealous (in a nice way).
One time I did it, I had been talking to one of the accountants at my work about my decision and he gave me this long spiel about how he had wife, kids, mortgage and had to suffer.
Then, 3 weeks later, I was working on a temping assignment in central London and on my lunch break, when I heard a guy calling me from a van. I turned round, and there was the accountant!
He said, "I thought to myself, if she can do it, so can I" and he had quit the week after me. He said "I'm earning half as much and I'm twice as happy driving this van".