The ability to recognize faces varies considerably between individuals, but does performance co-vary for tests of different aspects of face processing? For 397 participants (of whom the majority were university students) we obtained scores on the Mooney Face Test, Glasgow Face Matching Test (GFMT), Cambridge Face Memory Test (CFMT) and Composite Face Test. Overall performance was significantly correlated for each pair of tests, and we suggest the term f for the factor underlying this pattern of positive correlations. However, there were large variations in the amount of variance shared by individual tests: The GFMT and CFMT are strongly related, whereas the GFMT and the Mooney test tap largely independent abilities. We do not replicate a frequently reported relationship between holistic processing (from the Composite test) and face recognition (from the CFMT)—indeed, holistic processing does not correlate with any of our tests. We report associations of performance with digit ratio and autism-spectrum quotient (AQ), and from our genome-wide association study we include a list of suggestive genetic associations with performance on the four face tests, as well as with f.