Reference phantoms are widely applied to evaluate the radiation dose for external exposure. However, the frequently used reference phantoms are based on Caucasians. Dose estimation for Asians using a Caucasian phantom can result in significant errors. This study recruited 40 volunteers whose body sizes are close to the average Taiwanese population. Magnetic resonance imaging was performed to obtain the organ volume for construction of the Taiwanese reference man (TRM) and Taiwanese reference woman (TRW). The dose conversion coefficients (DCC) resulting from photo beams in anterior-posterior, posterior-anterior, right-lateral, left-lateral, and isotropic irradiation geometries were estimated. In the anterior-posterior geometry, the mean DCC differences among organs between the TRM and ORNL phantom at 0.1, 1, and 10 MeV were 7.3%, 5.8%, and 5.2%, respectively. For the TRW, the mean differences from the ORNL phantom at the three energies were 10.6%, 7.4%, and 8.3%. The DCCs of the Taiwanese reference phantoms and the ORNL phantom presented similar trends in other geometries. The torso size of the phantom and the mass and geometric location of the organ have a significant influence on the DCC. The Taiwanese reference phantoms can be used to establish dose guidelines and regulations for radiation protection from external exposure.