With the cultivation of genetically modified crops, transgenes may spread by introgression from crops into weedy and wild populations of related species. The likelihood of this depends in part on the fitness of first and later generation hybrids. We here present results on the fitness of F2 and backcross hybrids between oilseed rape (Brassica napus) and weedy B. rapa. Two populations of B. rapa, two varieties of B. napus, and their F1 hybrids were used for controlled crosses, and seed development, survival in the field, pollen viability, pod- and seed-set were estimated for the offspring. Offspring from F2 and backcrosses had a reduced fitness relative to their parents for most of the fitness components and for a combined estimate of fitness, with F2 offspring suffering the lowest fitness. Despite their lower fitness on average, some of the hybrids were as fit as the parents. Significant fitness differences were detected between backcross and F2 offspring from different B. rapa populations, B. napus varieties, and parental plants. Our results suggest that introgression of transgenes from oilseed rape to B. rapa will be slowed down, but not hindered, by the low fitness of second generation hybrids.