Working memory (WM) is a buffer that temporarily maintains information, be it visual or auditory, in an active state, caching its contents for online rehearsal or manipulation. How the brain enables long-term semantic knowledge to affect the WM buffer is a theoretically significant issue awaiting further investigation. In the present study, we capitalise on the knowledge about famous individuals as a ‘test-case’ to study how it impinges upon WM capacity for human faces and its neural substrate. Using continuous theta-burst transcranial stimulation combined with a psychophysical task probing WM storage for varying contents, we provide compelling evidence that (1) faces (regardless of familiarity) continued to accrue in the WM buffer with longer encoding time, whereas for meaningless stimuli (colour shades) there was little increment; (2) the rate of WM accrual was significantly more efficient for famous faces, compared to unknown faces; (3) the right anterior-ventrolateral temporal lobe (ATL) causally mediated this superior WM storage for famous faces. Specifically, disrupting the ATL (a region tuned to semantic knowledge including person identity) selectively hinders WM accrual for celebrity faces while leaving the accrual for unfamiliar faces intact. Further, this ‘semantically-accelerated’ storage is impervious to disruption of the right middle frontal gyrus and vertex, supporting the specific and causative contribution of the right ATL. Our finding advances the understanding of the neural architecture of WM, demonstrating that it depends on interaction with long-term semantic knowledge underpinned by the ATL, which causally expands the WM buffer when visual content carries semantic information.