One observer watches a recording and marks that a participant asked for clarification. A second watches the same clip and records a zero. If you are using observation in an education thesis, check how often disagreements like this occur. A worked example with 40 clips shows how to match the ratings, calculate percentage agreement and Cohen’s kappa, and report both in your thesis.
Start by pairing the ratings. Give both observers the same list of clips. Ask them to record their decisions separately and hand in their sheets before discussing differences. Use their original ratings for the calculations. You can keep the codes agreed on later in a separate column.
Decide what a single rating represents
Imagine a fictional education thesis on workshops for adults at a university in Poland. Two observers watch the same 40 clips, each 30 seconds long, with five clips from each of eight sessions. Code 1 means the clip includes at least one spoken request to clarify the task instructions. Code 0 means the clip could be assessed and included no such request. If the facilitator repeats the instructions unprompted, the rating is still 0. Two requests in a single clip still give a single rating of 1.
Record the session number, clip ID, start and end times and coding guide version. Join the two observers’ records by clip ID, so each row contains two ratings for the same clip. Check for duplicate IDs. If one observer cannot rate a clip because the recording is inaudible, mark the rating as missing. Decide how you will handle such pairs in advance. Report the total number of clips and the number with two valid ratings. Here all 40 pairs are complete.
Try the instructions on material you will keep out of the main agreement analysis. Suppose one observer marks a brief “How?” as 1 and the other marks it as 0. Clarify in the guide that such a question counts when the context shows it concerns how to do the task. Finalise the rules after the pilot and give both observers the same version. For the methods chapter, see “How to describe a research instrument in your thesis: an original questionnaire and an existing scale”.
Count how often the observers agreed
Label the observers A and B. In each pair, read A’s code first and then B’s. The 40 rows give you four counts.
- 1 and 1. Both heard a request in 8 clips.
- 1 and 0. Only A marked a request in 2 clips.
- 0 and 1. Only B marked a request in 3 clips.
- 0 and 0. Both recorded no request in 27 clips.
Add 8 and 27. The ratings match in 35 of the 40 pairs. That is 0.875, or 87.5%. The proportion of matching pairs is denoted by po. The four counts also show how often each observer used each code. A used 1 in 10 clips and 0 in 30. B used 1 in 11 clips and 0 in 29.
Kappa uses those proportions to set a benchmark. Imagine the observers assigning codes independently, with each using the codes as often as they did in the study. The expected proportion of matching ones is (10/40) × (11/40), and that of matching zeros is (30/40) × (29/40). Adding them gives pe = 0.6125. This model puts the actual matches in the context of how often each code was used.
Unweighted Cohen’s kappa. Subtract expected agreement from observed agreement. Divide the result by the gap between expected and perfect agreement. κ = (0.875 – 0.6125)/(1 – 0.6125) ≈ 0.68. In your results, report the coefficient of 0.68 alongside the agreement of 87.5%.
Look at what accounts for the matching ratings
In the first set, both observers identified a request in the same eight clips. Compare it with a second fictional set. In 38 of the 40 clips, both observers record zero. They each record a one once, in different clips. Percentage agreement is 95%, and kappa is about -0.03. Near-perfect agreement on the absence of an event can therefore produce a high percentage even when the observers never mark the event in the same clip. Jarosz-Nowak discusses how category frequencies affect kappa in her Polish-language paper.
If both observers record zero every time, percentage agreement is 100%. The model gives pe = 1, which puts zero in the denominator and makes kappa undefined. State the reason and report the pair counts. Very few ones give you limited evidence about how the instructions work for that category.
In the first set, return to the five clips with different ratings. For example, check whether several contain a brief “How?” with no follow-up. Agreement tells you how similarly the observers used the guide. Validity needs a separate argument. If you want to study task difficulty, consider whether a spoken request captures a participant who struggled and stayed silent.
Five clips from the same session may resemble one another more than clips from different sessions. The calculations describe the 40 clips selected. If you need a confidence interval for your thesis, choose a method that accounts for the relationship between clips from the same session. If you want to generalise to other sessions, also explain how you selected the material.
How to report agreement in your thesis
Two observers independently rated 40 clips, each 30 seconds long, from eight workshops. Code 1 meant that at least one person asked aloud for clarification of the task instructions. Code 0 meant that the clip could be assessed and contained no such request. All rating pairs were complete. Both observers recorded 1 in eight clips and 0 in 27. Their ratings differed in five clips (two pairs of 1 and 0 and three of 0 and 1). For the clips analysed, percentage agreement was 87.5% (35/40) and unweighted Cohen’s kappa was κ = 0.68.
What should readers find in your thesis?
- In the methods chapter, explain where the clips came from, what the codes mean and which version of the instructions the observers used.
- Explain how each observer rated independently, how many pairs were complete and how you handled missing ratings.
- In the results, report all four pair counts, the percentage agreement and unweighted kappa. If kappa is undefined, give the reason.
- In the discussion, say what you learnt from the disagreements and which material the calculation covers.
Questions about reporting the instrument and study limitations
What should a research instrument description in a thesis include?
State the instrument’s name and origin, purpose, language version, structure, response options and scoring rules. Explain its conditions of use, checks performed and limitations. Describe the questionnaire actually used rather than a revised version created afterwards.
How can I connect a limitation to its effect on the conclusions?
Identify a specific feature of the procedure, the finding or claim it affects, and the consequence. Then explain what you actually did to address it and what remains uncertain. A generic list mentioning sample size and time pressure is not enough.
References
- Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 37-46. DOI 10.1177/001316446002000104.
- Hallgren, K. A. (2012). Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23-34. DOI 10.20982/tqmp.08.1.p023.
- Hallgren, K. A. (2013). Erratum to Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 9(2), 95. DOI 10.20982/tqmp.09.2.p095.
- Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543-549. DOI 10.1016/0895-4356(90)90158-L.
- Jarosz-Nowak, J. (2007). Modele oceny stopnia zgody pomiędzy dwoma ekspertami z wykorzystaniem współczynników kappa. Matematyka Stosowana, 8, 126-154. DOI 10.14708/ma.v35i49.08.1391.
Refine your instructions or share the coded ratings
Tell us where you are in the process. Before the observations, we can help develop and describe the coding guide. Once you have two sets of ratings, we can check agreement and discuss how to report it.
Tell us about your observation study
