A sense of motion can be elicited by the movement of both luminance- and texture-defined patterns, what is commonly referred to as first- and second-order, respectively. Although there are differences in the perception of these two classes of motion stimuli, including differences in temporal and spatial sensitivity, it is debated whether common or separate direction-selective mechanisms are responsible for processing these two types of motion. Here, we measured direction-selective responses to luminance- and texture-defined motion in the human visual cortex by using functional MRI (fMRI) in conjunction with multivariate pattern analysis (MVPA). We found evidence of direction selectivity for both types of motion in all early visual areas (V1, V2, V3, V3A, V4, and MT+), implying that none of these early visual areas is specialized for processing a specific type of motion. More importantly, linear classifiers trained with cortical activity patterns to one type of motion (e.g., first-order motion) could reliably classify the direction of motion defined by the other type (e.g., second-order motion). Our results suggest that the direction-selective mechanisms that respond to these two types of motion share similar spatial distributions in the early visual cortex, consistent with the possibility that common mechanisms are responsible for processing both types of motion.