We investigated whether midlife pulse pressure is associated with brain atrophy and cognitive decline, and whether the association was modified by apolipoprotein-E ε4 (APOE-ε4) and hypertension. Participants (549 stroke-free and dementia-free Framingham Offspring Cohort Study participants, age range=55.0 to 64.9 y) underwent baseline neuropsychological and magnetic resonance imaging (subset, n=454) evaluations with 5- to 7-year follow-up. Regression analyses investigated associations between baseline pulse pressure (systolic−diastolic pressure) and cognition, total cerebral volume and temporal horn ventricular volume (as an index of smaller hippocampal volume) at follow-up, and longitudinal change in these measures. Interactions with APOE-ε4 and hypertension were assessed. Covariates included age, sex, education, assessment interval, and interim stroke. In the total sample, baseline pulse pressure was associated with worse executive ability, lower total cerebral volume, and greater temporal horn ventricular volume 5 to 7 years later, and longitudinal decline in executive ability and increase in temporal horn ventricular volume. Among APOE-ε4 carriers only, baseline pulse pressure was associated with longitudinal decline in visuospatial organization. Findings indicate arterial stiffening, indexed by pulse pressure, may play a role in early cognitive decline and brain atrophy in mid to late life, particularly among APOE-ε4 carriers.