This systematic review examined longitudinal associations between weight change (weight gain and loss) and both physical and mental aspects of health-related quality of life (HRQOL) compared with stable weight in adults and children of the general population. MEDLINE, EMBASE, PsycINFO and PubMed databases were searched. Longitudinal observational studies measuring HRQOL with six predefined instruments were synthesized according to type of association: weight change and change in HRQOL (change-on-change association) and weight change and HRQOL at follow-up (predictive association). Twenty studies of adults (n = 15) or children (n = 5) were included. Fifteen studies used the SF-12 or SF-36. Results of nine studies in adults examining the change-on-change association were combined through a tallying of 606 analyses. Weight gain was most often associated with reduced physical, but not mental HRQOL, across all baseline body mass index categories and in both men and women. Weight loss may be associated with improved physical, but not mental HRQOL, among adults with overweight and obesity. Weight gain was more strongly associated with HRQOL than weight loss, implicating a greater need for preventative strategies to tackle obesity. Results in children and for the predictive association generally reflected these findings but require further research.