This paper presents experimental results on polarizers with nonrectangular grooves to be used in high power millimeter wave transmission lines for electron cyclotron heating (ECH) of fusion plasmas. A low-power test at a frequency of 106.4 GHz has been carried out for several polarizers with different groove depths. The polarization characteristics experimentally measured are in good agreement with numerical results in which the actual groove shape functions are taken into account. This confirms the design optimization for future polarizer. Favorable results have also been obtained in a 400 kW-level high-power test for the polarizer installed in an ECH transmission line of the Heliotron-E plasma device.