Switching between mental sets has been extensively investigated in both experimental and individual differences research using a wide range of task-switch paradigms. However, it is yet unclear whether these different tasks measure a unitary shifting ability or reflect different facets thereof. In this study, 20 task pairs were administered to 119 young adults to assess 5 proposed components of mental set shifting: switching between judgments, stimulus dimensions, stimulus–response mappings, response sets, and stimulus sets. Modeling latent factors for each of the components revealed that a model with 5 separate yet mostly correlated factors fit the data best. In this model, the components most strongly related to the other latent factors were stimulus–response mapping shifting and, to a lesser degree, response set shifting. In addition, both factors were statistically indistinguishable from a second-order general shifting factor. In contrast, shifting between judgments as well as stimulus dimensions consistently required separate factors and could, hence, not fully be accounted for by the general shifting factor. Finally, shifting between stimulus sets was unrelated to any other shifting component but mapping shifting. We conclude that tasks assessing shifting between mappings are most adequate to assess general shifting ability. In contrast, shifting between stimulus sets (e.g., as in the Trail Making Test) probably reflects shifts in visual attention rather than executive shifting ability.