The perceived pitch of iterated rippled noise (IRN) with negative gain (IRNn) is an octave lower than that of IRN with positive gain (IRNp). IRNp and IRNn have identical waveform envelopes (ENV), but differing stimulus waveform fine structure (TFS), which likely accounts for this perceived pitch difference. Here, we examine whether differences in the temporal pattern of phase-locked activity reflected in the human brainstem Frequency Following Response (FFR) elicited by IRNp and IRNn can account for the differences in perceived pitch for the two stimuli. FFRs using a single onset polarity were measured in 13 normal-hearing, adult listeners in response to IRNp and IRNn stimuli with 2 ms, and 4 ms delay. Autocorrelation functions (ACFs) and Fast Fourier Transforms (FFTs) were used to evaluate the dominant periodicity and spectral pattern (harmonic spacing) in the phase-locked FFR neural activity. For both delays, the harmonic spacing in the spectra corresponded more strongly with the perceived lowering of pitch from IRNp to IRNn, compared to the ACFs. These results suggest that the FFR elicited by a single polarity stimulus reflects phase-locking to both stimulus ENV and TFS. A post-hoc experiment evaluating the FFR phase-locked activity to ENV (FFRENV), and TFS (FFRTFS) elicited by IRNp and IRNn confirmed that only the phase-locked activity to the TFS, reflected in FFRTFS, showed differences in both spectra and ACF that closely matched the pitch difference between the two stimuli. The results of the post-hoc experiment suggests that pitch-relevant information is preserved in the temporal pattern of phase-locked activity and suggests that the differences in stimulus ENV and TFS driving the pitch percept of IRNp and IRNn are preserved in the brainstem neural response. The scalp recorded FFR may provide for a noninvasive analytic tool to evaluate the relative contributions of envelope and temporal fine-structure in the neural representation of complex sounds in humans.