The epigenome is a dynamic mediator of gene expression that shapes the way that cells, tissues, and organisms respond to their environment. Initial studies in the emerging field of “toxicoepigenetics” have described either the impact of an environmental exposure on the epigenome or the association of epigenetic signatures with the onset or progression of disease; however, the majority of these pioneering studies examined the relationship between discrete epigenetic modifications and the effects of a single environmental factor. Although these data provide critical blocks with which we construct our understanding of the role of the epigenome in susceptibility and disease, they are akin to individual letters in a complex alphabet that is used to compose the language of the epigenome. Advancing the use of epigenetic data to gain a more comprehensive understanding of the mechanisms underlying exposure effects, identify susceptible populations, and inform the next generation risk assessment depends on our ability to integrate these data in a way that accounts for their cumulative impact on gene regulation. Here we will review current examples demonstrating associations between the epigenetic impacts of intrinsic factors, such as such as age, genetics, and sex, and environmental exposures shape the epigenome and susceptibility to exposure effects and disease. We will also demonstrate how the “epigenetic seed and soil” model can be used as a conceptual framework to explain how epigenetic states are shaped by the cumulative impacts of intrinsic and extrinsic factors and how these in turn determine how an individual responds to subsequent exposure to environmental stressors.