We investigated the specificity of face compared with object cognition from an individual differences and aging perspective by determining the amount of overlap between these abilities at the level of latent constructs across age. Confirmatory factor analytic models tested the specificity of speed and accuracy measures for face and object cognition (N = 448; 18 to 88 years). Accuracy measures were distinguishable and slightly dedifferentiated across age, which was not due to loss of visual acuity and contrast sensitivity. There was no face specificity for speed measures. These results support the specificity of face cognition from differential and developmental perspective only for performance accuracy.