Cerebral microbleeds (CMBs) have been identified as an important manifestation and diagnostic marker of cerebrovascular disease, but there are also some controversies about the impact of CMBs on cognition. The current cross-sectional study aimed to clarify the relationship between CMBs and cognitive impairment in patients with ischemic cerebrovascular disease. One hundred and fifty patients with lacunar infarction or transient ischemic attack were screened, and all of them were scanned by brain MRI. In the end, 125 patients who were divided into a CMBs group and a non-CMBs group were selected and completed the cognitive tests. Cognitive Function Assessment was performed using the Montreal Cognitive Assessment scale. Images of CMBs were assessed using a susceptibility-weighted imaging measure. Associations between cognitive function and the location of CMBs were determined. There was no significant difference in the demographic and clinical features between the two groups of patients. Compared with participants with no CMBs, the CMBs group showed a greater impairment in cognitive parameters and specifically in performance on three cognitive domains: visual space and executive function, memory, and abstract thinking. Basal ganglia–thalamic was associated with memory and visual space and executive function. Relationships between cortical–subcortical and abstract thinking became significant. Furthermore, the mixed region was related to memory, abstract thinking, and visual space and executive function. In summary, patients with CMBs had a greater impairment in cognitive parameters in ischemic cerebrovascular disease and CMBs location was associated with different cognitive parameters, adding to our understanding of the cognitive effects of CMBs in cerebrovascular disease.