Surface roughness of a sample has a great effect on the calculated grain size when measurements are based on ultrasonic attenuation. Combining modified transmission and reflection coefficients at the rough interface with a Multi-Gaussian beam model of the transducer, a comprehensive correction scheme for the attenuation coefficient is developed. An approximate inverse model of the calculated attenuation, based on Weaver's diffuse scattering theory, is established to evaluate grain size in polycrystals. The experimental results showed that for samples with varying surface roughness and matching microstructures, the fluctuation of evaluated average grain size was ±1.17 μm. For polished samples with different microstructures, the relative errors to optical microscopy were no more than ±3.61%. The presented method provides an effective nondestructive tool for evaluating the grain size in metals with rough surfaces.