The temporal lobe has been associated with various cognitive functions which include memory, auditory cognition and semantics. However, at a higher level of conceptualisation, all of the functions associated with the temporal lobe can be considered as lying along one major axis; from modality-specific to modality-general processing. This paper used a spectral reordering technique on resting-state and task-based functional data to extract the major organisational axis of the temporal lobe in a bottom-up, data-driven fashion. Independent parcellations were performed on resting-state scans from 71 participants and active semantic task scans from 23 participants acquired using dual echo gradient echo planar imaging in order to preserve signal in inferior temporal cortex. The resulting organisational axis was consistent (over dataset and hemisphere) and progressed from superior temporal gyrus and posterior inferior temporal cortex to ventrolateral anterior temporal cortex. A hard parcellation separated a posterior (superior temporal and posterior fusiform and inferior temporal gyri) and an anterior cluster (ventrolateral anterior temporal lobe). The functional connectivity of the hard clusters supported the hypothesis that the connectivity gradient separated modality-specific and modality-general regions. This hypothesis was then directly tested by performing a VOI analysis upon an independent semantic task-based data set including auditory and visually presented stimuli. This confirmed that the ventrolateral anterior aspects of the temporal lobe are associated with modality-general processes whilst posterior and superior aspects are specific to certain modalities, with the posterior inferior subregions involved in visual processes and superior regions involved in audition.