The aim of the present study was to investigate the age-related patterns and the relationships between serum levels of tartrate-resistant acid phosphatase-5b (TRACP-5b) or bone-specific alkaline phosphatase (BAP), and the heel stiffness index measured by quantitative ultrasound (QUS) in 429 Japanese men, with special emphasis on 2 age groups (40–59 years and 60 years or over). The heel stiffness index (bone mass) was measured by QUS. Serum samples were collected, and TRACP-5b and BAP levels were measured. The stiffness index was significantly decreased with age. Log (TRACP-5b) was significantly increased with age, but Log (BAP) was stable. Generalized linear models showed that higher levels of Log (TRACP-5b) and Log (BAP) were correlated with a lower stiffness index after adjusting for covariates in men aged 60 years or over, but not in men aged 40 to 59 years. In conclusion, higher rates of bone turnover markers were associated with a lower stiffness index only in elderly men. These results may indicate a different mechanism of low bone mass among different age groups of men.