The semantic variant of primary progressive aphasia (PPA-S) is diagnosed based on impaired single-word comprehension, but nonverbal impairments in face and object recognition can also be present, particularly in later disease stages. PPA-S is associated with focal atrophy in the left anterior temporal lobe (ATL), often accompanied by a lesser degree of atrophy in the right ATL. According to a dual-route account, the left ATL is critical for verbal access to conceptual knowledge while nonverbal access to conceptual knowledge depends upon the integrity of right ATL. Consistent with this view, single-word comprehension deficits in PPA-S have consistently been linked to the degree of atrophy in left ATL. In the current study we examined object processing and cortical thickness in 19 patients diagnosed with PPA-S, to evaluate the hypothesis that nonverbal object impairments would instead be determined by the amount of atrophy in the right ATL. All patients demonstrated inability to access conceptual knowledge on standardized tests with word stimuli: they were unable to match spoken words with their corresponding pictures on the Peabody Picture Vocabulary Test. Only a minority of patients, however, performed abnormally on an experimental thematic verification task, which requires judgments as to whether pairs of object pictures are thematically-associated, and does not rely on auditory or visual word input. The entire PPA-S group showed cortical thinning in left ATL, but atrophy in right ATL was more prominent in the subgroup with low verification scores. Thematic verification scores were correlated with cortical thickness in the right rather than left ATL, an asymmetric mapping which persisted when controlling for the degree of atrophy in the contralateral hemisphere. These results are consistent with a dual-route account of conceptual knowledge: breakdown of the verbal left hemispheric route produces an aphasic syndrome, which is only accompanied by visual object processing impairments when the nonverbal right hemispheric route is also compromised.