Advanced workshops of analysis methods - Inter-rater reliability

Teacher(s)

Coertjens Liesje;

Language

English

Main themes

Each year, a list of workshops related to "measurement" is provided to the students. These workshops address in detail the tools, methods and the specific analyzing techniques applied in research in Psychology and Educational Sciences. Each workshop has a duration of 15 hours. Students need to choose 2 different workshops for a total of 4 credits. Each workshop can only be attended once.

Content

Often in research in psychology and education, one relies on human judgement to assess, for example if an article should be included or excluded in a systematic literature review, the grade a certain answer or portfolio merits, or if a patient is at risk based upon multiple sources of data. Even when researchers try their very best to and are supported (for example with a checklist or rubric), raters are likely to disagree. This workshop focuses on defining inter-rater reliability, examining the degree of inter-rater reliability in a statistical way and developing pathways to increase inter-rater reliability if need be.

Teaching methods

This will be a workshop in small groups. There will be short lectures and exercises on real cases using statistical packages.
The course will be taught in English. 

Evaluation methods

The assessment consists of an individual project on real data. Students can choose to do this individual project in French or in English. More information will be provided during the first lesson. 

Bibliography

Gwet, K.L. (2014). Handbook of Inter-rater Reliability. Gaithersburg, MD: Advanced Analytics LLC.

Faculty or entity

EPSY