Extracting meaning from faces to understand other people's mental states and intentions, and to quickly adjust our actions accordingly, is a vital aspect of our social interactions. However, not all emotionally relevant attributes of a person are directly observable from facial features or expressions. In this study event-related brain potentials were used to investigate the effects of affective information about a person's biography that cannot be derived from the visual appearance of the face. Faces of well-known and initially unfamiliar persons with neutral expressions were associated with negative, positive or neutral biographical information. For well-known faces, early event-related brain potential (ERP) modulations induced by emotional knowledge, their scalp topographies and time course strongly resemble the effects frequently reported for emotional facial expressions even though here, access to stored semantic knowledge is required. These results demonstrate that visually opaque affective knowledge is extracted at high speed and modulates sensory processing in the visual cortex.