Rhythmic abilities vary widely in the general population, but little is known about the factors that give rise to this variability. One factor may be musical training. Another may be differences in auditory short-term memory (STM) capacity (the amount of auditory information that can be remembered over a few seconds). Finally, as rhythms with temporal regularity (e.g., a beat) are more easily remembered and reproduced, individual differences in sensitivity to regularity may contribute to rhythmic ability differences. To investigate the contribution of each of these factors to rhythm reproduction ability, we assessed auditory STM capacity (using digit and pseudoword span tasks), beat sensitivity (using the Beat Alignment Test [BAT]), and levels of musical training. Rhythmic ability was measured using a rhythm reproduction test. We found that STM capacity, beat sensitivity, and musical training predicted unique variance in rhythm reproduction performance. Using functional magnetic resonance imaging (MRI), we assessed individual differences in brain activity related to the previously measured auditory STM capacity, BAT score, musical training, and rhythmic ability, while participants performed a rhythm discrimination task. Activity in posterior superior temporal gyrus and middle temporal gyrus negatively correlated with auditory STM capacity. Positive correlations with BAT score were found in left angular gyrus, supplementary motor area, and premotor cortex. Positive correlations with musical training were observed in left posterior middle temporal gyrus, and negative correlations were observed in left supplementary motor area. The findings implicate both auditory and motor areas in factors that underlie individual differences in rhythmic ability.