A semi-parametric model for estimating hemodynamic response function (HRF) from multi-subject fMRI data is introduced within the context of the General Linear Model. The new model assumes that the HRFs for a fixed brain voxel under a given stimulus share the same unknown functional form across subjects, but differ in height, time to peak, and width. A nonparametric spline-smoothing method is developed to evaluate this common functional form, based on which subject-specific characteristics of the HRFs can be estimated. This semi-parametric model explicitly characterizes the common properties shared across subjects and is flexible in describing various brain hemodynamic activities across different regions and stimuli. In addition, the temporal differentiability of the employed spline basis enables an easy-to-compute way of evaluating latency and width differences in hemodynamic activity. The proposed method is applied to data collected as part of an ongoing study of socially mediated emotion regulation. Comparison with several existing methods is conducted through simulations and real data analysis.