To what extent is varying cognitive expertise reflected in the brain's functional specificity and connectivity? We addressed this question by examining expertise in mathematics based on the fact that mathematical skills are one of the most critical cognitive abilities known to be a good predictor of academic achievement. We investigated processing of hierarchical structures, which is a fundamental process for building complex cognitive architecture. Experts and nonexperts in mathematics participated in processing hierarchical structures using algebraic expressions. Results showed that a modulating effect depending on expertise was observed specifically in nonexperts in the left inferior frontal gyrus around pars triangularis and frontal sulcus, the left intraparietal sulcus, and the right inferior parietal lobule. This expertise-dependent pattern of activation led to a crucial dissociation within the left prefrontal cortex. More interestingly, task-related functional networks were also modulated differently in the frontoparietal network for relatively good performance and in the frontostriatal network for poor performance. The present study indicates that a high level of expertise is evident in a small number of specific brain regions, whereas a low level of expertise is reflected by broadly distributed brain areas, along with divergent functional connectivity between experts and nonexperts.