Despite the widespread application of periphytic diatoms to water quality assessment at a regional level, there is no standard European sampling protocol or associated assessment metrics. Furthermore, relatively little is known about the uncertainty in the results of such assessments. One of the objectives of the European project for the Standardisation of River Classifications (STAR) is to improve and standardise diatom assessment methods. An extensive diatom ring test, together with an audit of the project results, provided a better understanding and quantification of the uncertainty in quality assessment of running waters using diatoms. The variation in multimetric analysis shows that the choice of site and substrate for sampling, the inter-operator differences in diatom taxonomy and the counting techniques are the primary sources of uncertainty. To some extent, this variation also reveals the robustness of specific metrics in relation to the sources of uncertainty. Of the three most common substrate types tested (stone, macrophyte and sediment), macrophytes emerge as the most preferred substrate for diatom sampling when performing multimetric water quality assessment.