Time to complete two tests of manual dexterity, the 9-hole Peg Test and Grooved Pegboard Test, increases with advancing age. However, the adaptations responsible for the differences in pegboard times between middle-aged and older adults are largely unknown. Potential mechanisms include neuromuscular characteristics, cognitive function, and cutaneous sensation. To provide a tractable framework to address these gaps in knowledge, the purpose of the current study was to identify the latent variables underlying age-associated differences in time to complete the 9-hole and grooved pegboard tests. The approach involved an independent component analysis that identified associations between the two pegboard times for the two groups of participants with two to six secondary outcomes. The common association across three of the four conditions (two groups and two pegboard tests) was features derived from force-matching tasks requiring submaximal isometric contraction. In addition, there were significant associations for older adults between age, measures of cognitive function, and pegboard times. Nonetheless, the significant associations were unique for each age group and pegboard test. The results provide a framework for subsequent mechanistic studies to identify the adaptations underlying age-associated declines in manual dexterity.