The context or trial history of a task influences response efficiency in mixed paradigms based on cognitive control demands for task set selection. In the current study, the impact of context on prosaccade and antisaccade trials in single and mixed tasks was investigated with BOLD fMRI. Prosaccades require a look towards a newly appearing target, while antisaccades require cognitive control for prepotent response inhibition and generation of a saccade to the opposite location. Results indicated slower prosaccade reaction times and more antisaccade errors for switched than repeated or single trials, and slower antisaccade reaction times for single than mixed trials. BOLD activation was greater for the mixed than the single context in frontal eye fields and precuneus, while switch trials had greater activation than repeat trials in posterior parietal and middle occipital cortex. Greater antisaccade activation was observed overall in saccade circuitry, although effects were evident primarily for the mixed task when considered separately. Finally, an interaction was observed in superior frontal cortex, precuneus, anterior cingulate, and thalamus with strong responses for antisaccade switch trials in the latter two regions. Altogether this response pattern demonstrated the sensitivity of cognitive control to changing task conditions, especially due to task switching costs. Such context-specific differences highlight the importance of trial history when assessing cognitive control.