Moving Towards a Mixed-Method Approach to Educational Assessments
In real-world assessments, numerous sources of information must be integrated. Just as in research, quantitative and qualitative methods in assessment complement one another. In the revised medical curriculum at my institution, we are adopting mixed-methods assessments as part of the assessment program. Guided by Miller’s2 pyramid of clinical competence, assessment methods and tools are selected based on appropriateness, with a focus on validity, reliability, educational impact, feasibility, and acceptability. For example, to assess competence at the “Knows” (basic knowledge) and “Knows How” (applied knowledge) levels, we opted for quantitative assessment, which is objective and measurable, using methods such as multiple-choice questions whereby content validity can be ensured by blueprinting while test item statistics and test score reliability are easily computed. To assess competence at the “Shows How” level (hands-on), objective structured clinical examinations (OSCEs) are preferred. OSCEs can assess clinical skills, communication skills, and professionalism/ethics. The OSCE marking sheet contains checklists and global ratings. For each checklist, students are scored according to three levels of competence (numeric scores) as “Performed/Appropriate” (Level 3), “Attempted Inappropriate” (Level 2), or “Not Attempted” (Level 1). Based on scores from the checklists, students are given a global rating as “Very Good,” “Satisfactory,” “Borderline,” or “Unsatisfactory.” Examiners also give written comments mentioning specific strengths and weaknesses (narrative data) on the marking sheet. Such triangulation of data sources is particularly important for “Borderline” or “Unsatisfactory” cases. Assessment of “Does” in Miller’s pyramid (students’ actual performance) remains challenging. Currently, evidence of satisfactory completion of an e-logbook comprising case clerking, case presentation, case summary, reflective essay, and procedural skills is part of the requirement to pass each clinical rotation.
Good decisions can be based on numbers or words but ideally both. Quantitative data supported by narrative data will make the assessment decisions more valid and defensible, especially in cases of appeal.