Several strategies have been proposed to model and remove physiological noise from resting-state fMRI (rs-fMRI) data, particularly at ultrahigh fields (7 T), including contributions from respiratory volume (RV) and heart rate (HR) signal fluctuations. Recent studies suggest that these contributions are highly variable across subjects and that physiological noise correction may thus benefit from optimization at the subject or even voxel level. Here, we systematically investigated the impact of the degree of spatial specificity (group, subject, newly proposed cluster, and voxel levels) on the optimization of RV and HR models. For each degree of spatial specificity, we measured the fMRI signal variance explained (VE) by each model, as well as the functional connectivity underlying three well-known resting-state networks (RSNs) obtained from the fMRI data after removal of RV+HR contributions. Whole-brain, high-resolution rs-fMRI data were acquired from twelve healthy volunteers at 7 T, while simultaneously recording their cardiac and respiratory signals. Although VE increased with spatial specificity up to the voxel level, the accuracy of functional connectivity measurements improved only up to the cluster level, and subsequently decreased at the voxel level. This suggests that voxelwise modeling over-fits to local fluctuations with no physiological meaning. In conclusion, our results indicate that 7 T rs-fMRI connectivity measurements improve if a cluster-based physiological noise correction approach is employed in order to take into account the individual spatial variability in the HR and RV contributions.