The consumption of psychoactive drugs during pregnancy can have deleterious effects on newborns. It remains unclear whether early-life exposure to caffeine, the most widely consumed psychoactive substance, alters brain development. We hypothesized that maternal caffeine ingestion during pregnancy and the early postnatal period in mice affects the construction and activity of cortical networks in offspring. To test this hypothesis, we focused on primary visual cortex (V1) as a model neocortical region. In a study design mimicking the daily consumption of approximately three cups of coffee during pregnancy in humans, caffeine was added to the drinking water of female mice and their offspring were compared to control offspring. Caffeine altered the construction of GABAergic neuronal networks in V1, as reflected by a reduced number of somatostatin-containing GABA neurons at postnatal days 6–7, with the remaining ones showing poorly developed dendritic arbors. These findings were accompanied by increased synaptic activity in vitro and elevated network activity in vivo in V1. Similarly, in vivo hippocampal network activity was altered from the neonatal period until adulthood. Finally, caffeine-exposed offspring showed increased seizure susceptibility in a hyperthermia-induced seizure model. In summary, our results indicate detrimental effects of developmental caffeine exposure on mouse brain development.