Smooth pursuit eye movements (SPEM) are used to maintain focus upon moving targets. The generation of SPEM velocity is controlled by retinal information and extraretinal signals. Although there is a wealth of studies investigating retinal and extraretinal SPEM control, the main questions regarding the cortical mechanisms involved in the processing of SPEM to different stimulus velocities are still unresolved. We applied an innovative event-related fMRI-design by presenting target ramps at different velocities (5, 10, 15, 20°/s) with both continuous target presentation and intervals of target blanking. The stimulus parameters were integrated into the statistical model and eye movements were registered to confirm SPEM performance. Our results clearly demonstrate that in humans the oculomotor network (V5, frontal and supplementary eye fields, lateral intraparietal area) is engaged in the processing of retinal and extraretinal SPEM velocity. Within this network neural activity increases with increasing target velocity. During extraretinal SPEM, additional engagement of the dorsolateral prefrontal cortex, angular gyrus, parahippocampal gyrus and superior temporal gyrus occurs. These regions encode cognitive functions such as memory, attention and monitoring. The activation of the inferior parietal cortex seems to be related to the interaction between velocity and blanking thereby underlining its relevance for task switching and sensorimotor transformation.