Objective Structured Assessment of Technical Skills: A Tool to Document Cystoscopic Competency in the Operating Room

Abstract

To demonstrate the construct and predictive validity of an objective structured assessment of technical skills (OSATS) checklist for cystoscopy in the operating room.

The OSATS checklist is a possible tool for documenting residency proficiency in cystoscopy.

Prospective observational study evaluating residents performing cystoscopy in the operating room over a 4-year period. Residents were scored on two OSATS checklists: assembly and diagnostic cystoscopy. Construct validity was assessed by comparing absolute OSATS scores as well as comparing the percentage of competent residents based on resident year with a Kruskall-Wallis and χ2 test, respectively. Predictive validity was evaluated by determining the percent of competently performed OSATS checklists subsequent to one competent OSATS completion.

One hundred fifty three OSATS evaluations were performed on 30 residents from all 4 years. The median scores and competency scores for assembly and performance of a diagnostic cystoscopy OSATS Checklist scores demonstrated construct validity with P<.001 and P<.0001, respectively. Once competency was obtained for one cystoscope assembly OSATS checklist, 92% of future assembly OSATS were documented as competent. For diagnostic cystoscopy once competency was obtained, 69% of future OSATS checklists were competent.

The OSATS checklist for assembly and diagnostic cystoscopy demonstrated construct validity. Predicative validity for assembly of a cystoscope is high but less so for performance of a diagnostic cystoscope. This checklist could be used to document proficiency in cystoscopy as part of a milestones portfolio.