OC-021 Changes in scoring of direct observation of procedural skills (dops) forms in endoscopy training and their impact on competence assessment

    loading  Checking for direct PDF access through Ovid

Abstract

Introduction

DOPS are validated tools for assessing competence in endoscopy. Previously, DOPS were scored on a 4-point competence-based scale, with scores of 3 and 4 signifying competence. In July 2016, the DOPS rating scale changed to a supervision-based scale that has been shown to be more reliable, [1] with 4 ratings from maximal supervision, up to competent without supervision. We aimed to assess whether changes to the rating scale have affected distribution of scores and hence demonstrate validity.

Method

Scores were collected from DOPS for gastroscopy (1934), sigmoidoscopy (517), colonoscopy (2296) and polypectomy (370) in the 6 months before July 2016 (old DOPS) and after (new DOPS), for trainees at early stages of training (total procedures<100). To allow analysis, the new DOPS rating scales was aligned to a 4-point scale, hence a score of 4 on new DOPS = Scores 3 or 4 on old DOPS, and scores on the new and old DOPS compared using the Mann-Whitney U-test.

Results

5117 DOPS (77.7% new and 22.3% old) were included for analysis. Overall, there were variations in distributions of scores (p<0.001) between forms (Graph). Compared to new DOPS, scores of 1 were underutilised on old DOPS (0.6% vs 3.0%, p<0.001). Frequencies of low scores (pooled scores of 1 and 2) were similar for gastroscopy (p=0.53) and sigmoidoscopy (p=0.34), but not for colonoscopy (new 11.9% vs. old 13.9%, p<0.001) and polypectomy (new 6.8% vs.19.9%, p<0.001). Trainees on old DOPS were more likely to be rated as competent (score 3 or 4) compared to new DOPS (86.4% vs. 55.8%, p<0.001). On subgroup analysis, this was evident for gastroscopy (86.3% vs. 49.1%, p<0.001), colonoscopy (86.1% vs. 58.2%, p<0.001), sigmoidoscopy (90.6% vs. 62.0%, p<0.001), but not polypectomy (80.1% vs. 67.9%, p=0.12).

Conclusion

Endoscopy assessors are applying a greater range of scores using a new DOPS rating scale based on degree of supervision, in two cohorts of trainees matched for experience. This indicates better construct validity with the new rating scale. Further work is underway to determine the reliability of the new DOPS to inform summative assessment and certification for UK endoscopy trainees.

Disclosure of Interest

None Declared

Related Topics

    loading  Loading Related Articles