Moving Towards a Mixed-Method Approach to Educational Assessments

    loading  Checking for direct PDF access through Ovid

Excerpt

The article by Cook et al1 is both enlightening and thought-provoking. While reminding readers that quantitative scores are inadequate to capture important features of learners’ performance, the authors advocate increased use of qualitative assessments only when warranted, and caution that the implementation of qualitative assessment methods requires training of assessors and analysts. The authors’ illustration of the use of a contemporary validity framework when evaluating any assessment (quantitative or qualitative), though it employs a hypothetical example, is most illuminating. In fact, it may be more desirable to adopt a mixed-methods approach to assessment using a combination of quantitative and qualitative methods.
In real-world assessments, numerous sources of information must be integrated. Just as in research, quantitative and qualitative methods in assessment complement one another. In the revised medical curriculum at my institution, we are adopting mixed-methods assessments as part of the assessment program. Guided by Miller’s2 pyramid of clinical competence, assessment methods and tools are selected based on appropriateness, with a focus on validity, reliability, educational impact, feasibility, and acceptability. For example, to assess competence at the “Knows” (basic knowledge) and “Knows How” (applied knowledge) levels, we opted for quantitative assessment, which is objective and measurable, using methods such as multiple-choice questions whereby content validity can be ensured by blueprinting while test item statistics and test score reliability are easily computed. To assess competence at the “Shows How” level (hands-on), objective structured clinical examinations (OSCEs) are preferred. OSCEs can assess clinical skills, communication skills, and professionalism/ethics. The OSCE marking sheet contains checklists and global ratings. For each checklist, students are scored according to three levels of competence (numeric scores) as “Performed/Appropriate” (Level 3), “Attempted Inappropriate” (Level 2), or “Not Attempted” (Level 1). Based on scores from the checklists, students are given a global rating as “Very Good,” “Satisfactory,” “Borderline,” or “Unsatisfactory.” Examiners also give written comments mentioning specific strengths and weaknesses (narrative data) on the marking sheet. Such triangulation of data sources is particularly important for “Borderline” or “Unsatisfactory” cases. Assessment of “Does” in Miller’s pyramid (students’ actual performance) remains challenging. Currently, evidence of satisfactory completion of an e-logbook comprising case clerking, case presentation, case summary, reflective essay, and procedural skills is part of the requirement to pass each clinical rotation.
Good decisions can be based on numbers or words but ideally both. Quantitative data supported by narrative data will make the assessment decisions more valid and defensible, especially in cases of appeal.
    loading  Loading Related Articles