Admixture can break up divergent genetic architectures between populations, resulting in phenotypic novelty and generating raw material for environmental selection. The contribution of admixture to progeny trait variation and fitness varies based on the degree of genetic isolation between the parental populations, for which most studies have used geographic distance as a proxy. A novel approach is to estimate optimal crossing distance using the adaptive genetic distance between mates estimated from loci that contribute directly to local adaptation. Here, we aim to understand the effect of admixture on disrupting local adaptation of ecotypes of Arabidopsis thaliana separated along gradients of geographic, background, and locally adaptive genetic distances. We created experimental F1 hybrids between ecotypes that vary in geographic distance and used SNP data to estimate background (putatively neutral) and adaptive genetic distance. Hybrids were grown under controlled conditions, and fitness, growth, and phenology traits were measured. The different traits measured showed a clear effect of adaptive genetic distance, but not geographic distance. The earliest bolting hybrids were intermediate in the adaptive genetic distance between their parents, and also had higher biomass and fitness in terms of fruit and seed production. Our results suggest that disruption of locally adaptive genomic loci decreases the performance of offspring between distantly related parents, but that crosses between very closely related parents also reduce performance, likely through the expression of deleterious recessive alleles. We conclude that during admixture, selection may have to balance the consequences of disrupting local adaption while also avoiding inbreeding depression.