Open Access

Sabine Ramspott, Ulrike Sonntag, Anja Härtl, Stefan Rüttermann, Doris Roller, Marianne Giesler, Linn Hempel

• Aim: The aim of the study was to develop a scoring rubric that provides valuable feedback to students and to gather evidence for its construct validity. Methodology: The Magdeburg Reflective Writing Feedback and Scoring Rubric (MaReS) was developed in an iterative process following a symposium on reflection by a committee of the "DACH Association for Medical Education (GMA)" in June 2016. 25 essays written by 13 veterinary students were assessed by three independent raters with MaReS and by two raters with the REFLECT rubric in two runs (13 and twelve essays). Validity evidence was gathered referring to the following of Messick's components of construct validity: content (rubric development), response process (rater manual, rater training, rating time, students' evaluation), internal structure (inter-rater reliability, IRR), and relationship to other variables (comparison of the rating with the REFLECT rubric and a global rating scale). Results: The analytic rubric comprises twelveAim: The aim of the study was to develop a scoring rubric that provides valuable feedback to students and to gather evidence for its construct validity. Methodology: The Magdeburg Reflective Writing Feedback and Scoring Rubric (MaReS) was developed in an iterative process following a symposium on reflection by a committee of the "DACH Association for Medical Education (GMA)" in June 2016. 25 essays written by 13 veterinary students were assessed by three independent raters with MaReS and by two raters with the REFLECT rubric in two runs (13 and twelve essays). Validity evidence was gathered referring to the following of Messick's components of construct validity: content (rubric development), response process (rater manual, rater training, rating time, students' evaluation), internal structure (inter-rater reliability, IRR), and relationship to other variables (comparison of the rating with the REFLECT rubric and a global rating scale). Results: The analytic rubric comprises twelve items that are rated on three-point rating scales. The authors developed an assignment with guiding questions for students and a rater manual. Results for free marginal kappa of the items of MaReS ranged from -0.08 to 0.77 for the first set of reflective essays and from 0.13 to 0.75 for the second set. Correlations between MaReS and the REFLECT rubric were positive (first run: r=0.92 (p<0.001); second run: r=0.29 (p=0.37)). Conclusion: MaReS might be a useful tool to guide students' reflective writing and provide structured feedback in health professions education. Using more essays for a rater training and more training cycles are likely to result in higher IRRs.…