This study investigates shear wave phase map reconstruction using a limited number of color flow images (CFIs) acquired with a color Doppler ultrasound imaging instrument. We propose an efficient reconstruction method to considerably reduce the number of CFIs required for reconstruction and compare this method with Fourier analysis-based color Doppler shear wave imaging. The proposed method uses a two-step phase reconstruction process, including an initial phase map derived from four CFIs using an advanced iterative algorithm of optical interferometry. The second step reduces phase artifacts in the initial phase map using an iterative correction procedure that cycles between the Fourier and inverse Fourier domains while imposing directional filtering and total variation regularization. We demonstrate the efficacy of this method using synthetic and experimental data of a breast phantom and human breast tissue. Our results show that the proposed method maintains image quality and reduces the number of CFIs required to four; previous methods have required at least 32 CFIs to achieve equivalent image quality. The proposed method is applicable to real-time shear wave elastography using a continuous shear wave produced by a mechanical vibrator.