Recent research suggests the existence of a visual area selectively processing numbers in the human inferior temporal cortex (number form area (NFA); Abboud et al., 2015; Grotheer et al., 2016; Shum et al., 2013). The NFA is thought to be involved in the preferential encoding of numbers over false characters, letters and non-number words (Grotheer et al., 2016; Shum et al., 2013), independently of the sensory modality (Abboud et al., 2015). However, it is not yet clear if this area is mandatory for normal number processing. The present study exploited the fact that high-resolution fMRI can be applied to identify the NFA individually (Grotheer et al., 2016) and tested if transcranial magnetic stimulation (TMS) of this area interferes with stimulus processing in a selective manner. Double-pulse TMS targeted at the right NFA significantly impaired the detection of briefly presented and masked Arabic numbers in comparison to vertex stimulation. This suggests the NFA to be necessary for fluent number processing. Surprisingly, TMS of the NFA also impaired the detection of Roman letters. On the other hand, stimulation of the lateral occipital complex (LO) had neither an effect on the detection of numbers nor on letters. Our results show, for the first time, that the NFA is causally involved in the early visual processing of numbers as well as of letters.