Wafer fabrication for semiconductor manufacturing consists of multiple layers, in which the displacements (i.e., overlay errors) between layers should be reduced to enhance the yield. Although it can reduce variance between layers by fixing the exposure machine (i.e. steeper or scanner), it is not practical to expose the wafer on the same machine from layer to layer for the lengthy fabrication process in real setting. Thus, there is a critical need to determine the similarity machine subgroups, in which appreciate backups for unexpected machine down can be also prioritized. This study aims to develop a novel methodology to fill this gap based on the proposed similarity measurement of systematic overlay errors and residuals. The proposed methodology was validated via empirical study in a wafer fab and the results showed practical viability of this approach.