In the present study, we investigated how overall and specific domains of physical activity and sedentary behavior at the age of 7 years were associated with cognition at the age of 11 years in 8,462 children from the Millennium Cohort Study. Data were collected from 2001 to 2013. Participation in domains of physical activity and sedentary behavior at 7 years of age were reported. Activity levels were also measured objectively. Cognition was assessed using the British Ability Scales. General linear models were used to assess longitudinal associations of physical activity and sedentary behavior, measured both objectively and via self-report, with cognition. Analyses were adjusted for prespecified covariates. Sports/physical activity club attendance (B = 0.6, 95% confidence interval (CI): 0.2, 1.1), doing homework (B = 0.5, 95% CI: 0.0, 0.9), and objectively measured sedentary time (B = 0.8, 95% CI: 0.1, 1.4) at age 7 years were positively associated with cognition at age 11 years in final the models. Television viewing was negatively associated with cognition (B = −1.7, 95% CI: −2.4, −1.0), although the association was attenuated to the null after adjustments for baseline cognition. Objectively measured light physical activity was inversely associated with cognition (B = −0.7, 95% CI: −1.3, −0.1). Moderate-to-vigorous physical activity was also inversely associated with cognition in girls only (B = −1.1, 95% CI: −2.0, −0.3). Associations of physical activity and sedentary behavior with cognition appear to be context-specific in young people.