I'm a "hardgainer" too and the book "brawn" by Stuart McRobert helped me quite a bit.
Amazon.com: Brawn, 3rd Edition (9789963916313): Stuart McRobert: Books
The gist of it is:
1. Add weight in small increments per week. Even if you add 2 lb a week that's a 104 lb in a year more than you were doing. He says that for "hard gainers" its easier to plateau than for "easy gainers." This seemed to hold true for me.
2. Even if you mainly want upper body strength/size you need to do squats or dead lifts because they seem to help trigger overall growth. Overall focus on the big combination lifts like bench presses, dead lifts, squats and rows or if you can chin ups with weight.
3. Eat like nothing else.
4. Cut out cardio until you've got the size/strength you want.