Hypertension has been related to structural and functional brain changes. In high-risk populations, hypertensive target organ damage might better represent exposure to high blood pressure than the blood pressure measurement itself. We examined the association of hypertensive target organ damage with longitudinal changes in brain structure and function within the Second Manifestations of Arterial Disease–Magnetic Resonance (SMART-MR) study. Renal function, albuminuria, and left ventricular hypertrophy on electrocardiography were measured in 663 patients with manifest arterial disease (mean age, 57±9 years; 81% men). Automated brain segmentation was used to quantify progression of global brain atrophy (change in brain parenchymal fraction) and progression of cerebral small vessel disease on 1.5T magnetic resonance imaging, and memory and executive functioning were assessed at baseline and after on average 3.9 years of follow-up. Regression analyses showed that an increasing number of signs of target organ damage was associated with more progression of global brain atrophy and more rapid decline in memory performance. Compared with no target organ damage, mean differences in change in brain parenchymal fraction (95% confidence interval) for 1 and ≥2 signs of organ damage were −0.12 (−0.30; 0.06) and −0.41 (−0.77; −0.05) % intracranial volume, and mean (95% confidence interval) differences in change in memory performance (z score) were −0.15 (−0.29; −0.00) and −0.27 (−0.54; −0.01). Results were independent of blood pressure, antihypertensive treatment, and other confounders. Hypertension target organ damage was not associated with progression of cerebral small vessel disease or change in executive functioning. Routinely assessed signs of hypertensive target organ damage, and in particular impaired renal function, could be used to identify patients at the highest risk of cognitive decline.