A 2-year field experiment was conducted (i) to study the effect of combined P and Zn fertilization on the Zn nutritional quality of wheat grown on potentially Zn-deficient calcareous soil and (ii) to identify which soil Zn fraction had the greatest effect on grain Zn concentration. Results showed that Zn fertilization increased grain Zn concentrations by 13% in 2006–2007 and 15% in 2007–2008. However, the application of 200 kg P2O5 ha−1 combined with Zn fertilizer reduced grain Zn concentrations by 38% in 2006–2007 and 17% in 2007–2008 compared with the control (no fertilizer treatment). The phytic acid (PA) concentration and the PA:Zn molar ratio in wheat grain increased as P fertilizer application rate increased, regardless of the Zn fertilizer application rate. Zinc fertilization alone increased diethyltriaminepentaacetic acid–Zn by 220% in 2006–2007 and 470% in 2007–2008 compared with the no Zn fertilizer treatment. The combined application of P and Zn also increased diethyltriaminepentaacetic acid–Zn and loose organic matter–bound Zn, but the increase became smaller as the P fertilizer application rate increased. In conclusion, P fertilizer application rates to the potentially Zn-deficient calcareous soil used in this study should be less than 100 kg P2O5 ha−1 to ensure the efficacy of the Zn fertilizer. Among the soil Zn fractions in this study, loose organic matter–bound Zn had the greatest correlation with grain Zn concentration.