Although pica is one of the most prominent signs in individuals with severe cognitive impairment, the mechanisms and neural basis for pica have not been well elucidated. To address this issue, patients with acquired brain injury who showed pica and hyperorality were investigated. Eleven patients with pica, i.e., individuals who eat non-food items, and eight patients with hyperorality but who never eat non-food items were recruited. The cognitive and behavioral assessments and neural substrates of the two groups were compared. For basic cognitive and behavioral functions, two kinds of mental state examination—the mini-mental state examination and the new clinical scale for rating of mental states of the elderly—were administered. For pica-related behavioral features, frontal release signs, semantic memory deficits, and changes in eating behaviors were compared. Compared with the hyperorality group, the pica group had more severe semantic memory deficits and fewer frontal release signs, whereas there was no significant difference in changes in eating behaviors. Individuals in the pica group always had a lesion in the posterior part of the middle temporal gyrus. These findings suggest that semantic memory deficits following temporal lobe damage are associated with pica.