Effective horse–rider interactions are a key component of good horse riding regardless of equestrian discipline, but they remain poorly understood in terms of quantifying. The aim of this study was to assess and describe factors of interaction in the rider–horse system at different skill levels for walk, sitting trot, and canter based on the inertial measurement technique. Horse–rider interaction was defined in terms of the time lag of cross-correlation between specific parameters of rider and horse: the smaller the time lag, the better the horse–rider interaction. Ten high level professional riders (PROs) and 10 beginners (BEGs) participated in this study. Based on simultaneous acceleration data and segment angles of rider’s segments (pelvis and trunk) and horse’s trunk, cross-correlation analysis was used to quantitatively evaluate the interaction for the factors gaits and skill level. The results indicate significant differences in interaction in the various equine gaits. Furthermore, the results of cross-correlation analysis suggest a better horse–rider interaction in roll (sagittal plane) than in pitch (frontal plane), regardless of the investigated skill levels and gaits. However, no significant differences were found between the two examined skill levels after multivariate analysis of variance (Bonferroni post hoc test). The factor skill level showed only statistical tendencies between the interaction of horse’s trunk and the rider’s pelvis. This study demonstrates the potential of a novel method to determine and characterize the interaction between horse and rider under field conditions. Such quantitative information can be very useful for judges, horsemen, and trainers.