Cannabis use in youth is rising and has been linked to deficits in cognitive functioning. However, cognitive findings have primarily been based on small samples of users seeking treatment, and few studies have evaluated cognition in occasional cannabis users. Here, we examined 4,568 adolescents and young adults (ages 14–21 years) drawn from the Philadelphia Neurodevelopmental Cohort, a prospective, population-based study. Participants were classified as cannabis Non-Users (n = 3,401), Occasional Users (twice per week or less; n = 940), or Frequent Users (>3 times per week; n = 227). Mixed-model analyses examined main effects of cannabis use and interactions between age and cannabis use on cognitive functioning. There was a significant interaction between cannabis group and age such that adolescent (but not young adult) Frequent Users performed worse than Non-Users on measures of executive control (p = .002). Earlier age of cannabis use was associated with worse performance in executive control in Occasional Users (p = .04). Unexpectedly, Occasional Users exhibited better executive control, memory, and social cognition than Non-Users (ps < .05). Although mild executive control deficits in adolescent frequent users and a relation between early cannabis initiation and cognitive performance are partially consistent with prior research, cognitive deficits were not found in other hypothesized domains in this community-based sample. Moreover, occasional cannabis users displayed equivalent or even slightly better executive control, social–cognitive, and memory abilities compared with nonusers, suggesting complex relationships between cannabis use and cognition in youth. Longitudinal studies with community samples are needed to identify variables affecting risk and resilience to cognitive deficits associated with cannabis.