The transition from childhood to adolescence is marked by a dramatic decrease in physical activity (PA). Physical education (PE) experiences may contribute to this change but remain underresearched. Using a retrospective survey, we examined whether memories of enjoyment or nonenjoyment of PE relate to present-day (adult) attitudes, intentions, PA, and sedentary behavior. An online questionnaire was completed by 1028 American respondents (18–45 yr). The participants rated their retrospective enjoyment of PE, present attitudes and intentions for PA, as well as present PA and sedentary behavior. In addition, participants responded with their best and worst PE memories in an open-ended fashion. Retrospective enjoyment of PE was associated with present-day attitude (r = 0.37, P < 0.00001) and intention (r = 0.23, P < 0.00001) for PA, as well as negatively associated with sedentary time on the weekend (r = −0.14, P < 0.00001). The best memories related to enjoyment of the activities in class (56%), experiencing feelings of physical competence (37%), and, interestingly, 7% were not having to take PE class any longer or skipping the class. Of the worst memories, 34% related to embarrassment, 18% to lack of enjoyment, 17% to bullying, 14% to social–physique anxiety, 16% to injury, and 2% to being punished by the PE teacher. Childhood memories of PE are associated with PA attitude, intention, and sedentary behavior in adulthood. Intensified research efforts should be directed toward understanding the factors and processes that lead to the formation of memories of PE.