A change from cultivated land to grassland generally increases soil organic matter (SOM) content and is a potential option to mitigate greenhouse gas emissions. We investigated the effects of two-year perennial grass and mixed grass/legume leys in a six-year crop rotation on topsoil (0–0.25 m depth) carbon content and on grain yields of winter wheat over a period of 31 years. Different nitrogen fertilisation regimes were included and no manure was added to the experimental plots. We used data from long-term crop rotation experiments at three sites in southern Sweden: Säby (59°49′ N/17°42′ E), Lanna (58°20′ N/13°07′ E) and Stenstugu (57°36′ N/18°26′ E). At Säby, the reduction in topsoil carbon content was smaller in the ley crop rotations than in the crop rotation with only annual crops. There were no statistically significant effects of crop rotation on topsoil carbon at the other two sites. At Lanna, the grain yield increase in winter wheat over time was higher in the mixed legume/grass ley crop rotation than in the other two rotations. Together, these effects of ley on topsoil carbon and winter wheat yield suggest that replacing annual crops with leys in the crop rotation could reduce losses of soil carbon without decreasing total yield of annual crops on a regional scale. We also applied the Introductory Carbon Balance Model (ICBM) to simulate topsoil carbon content at the three sites. Based on the results, measures to improve the model predictability are proposed.