The American Academy of Orthopaedic Surgeons Appropriate Use Criteria for Surgical Management of Osteoarthritis of the Knee
Members of the Writing Panel developed a list of 864 patient scenarios based on the identified patient indications (Table I) and defined sub-classifications along with 3 treatments (Table II), for a total of 2,592 total voting items (864 scenarios × 3 treatments).
The Writing Panel reviewed these scenarios and treatments to ensure that they were representative of patients and scenarios clinicians are likely to encounter. Each independent panel member was given the opportunity to suggest potential modifications to the content or structure of the Writing Panel materials. The Writing Panel provided final determination of modifications to the indications, scenarios, assumptions, and literature review.
The Voting Panel participated in 3 rounds of voting. During the first round, the Voting Panel was given approximately 2 months to independently rate the appropriateness of the provided treatments for each of the relevant patient scenarios as “Appropriate,” “May Be Appropriate,” or “Rarely Appropriate” via an electronic ballot (Table III). After the first round of appropriateness ratings were submitted, AAOS staff calculated the median appropriateness ratings for each treatment and patient scenario as well as the level of disagreement between the panel members. The Voting Panel then met at an in-person meeting in Rosemont, Illinois, on Friday, August 5, 2016, to discuss any scenarios/treatments that had resulted in disagreement during the first round of voting. The Voting Panel members discussed the list of assumptions, patient indications, and treatments to identify areas that needed to be clarified and/or edited. Following the discussion and subsequent changes, members of the Voting Panel had the option to change any of their appropriateness ratings during the second round of electronic voting. After the completion of a second round of voting, the Voting Panel opted to revisit the scenarios about which there was still disagreement and opened the electronic ballot for a third round of voting. There was no attempt to obtain consensus regarding appropriateness.
With use of the median value of the second-round ratings from the members of the Voting Panel, the final levels of appropriateness were determined. Disagreement among raters can affect the final rating. Agreement and disagreement were determined with use of the BIOMED definitions of Agreement and Disagreement as reported in the RAND/UCLA Appropriate Method User’s Manual2 for a panel of 11 to 13 voting members. For this panel size, agreement was defined as when ≤3 panelists rated outside of the 3-point range containing the median and disagreement was defined as when ≥4 members’ appropriateness ratings fell within the “Appropriate” (7 to 9) and “Rarely Appropriate” (1 to 3) ranges for any scenario. If there was still disagreement in the Voting Panel ratings after the second round of voting, that voting item was labeled as “5” regardless of the median score.