The intraclass correlation coefficient (ICC) measures how consistently raters, or repeated measurements, score the same things. It is the standard statistic for inter-rater and test-retest reliability of continuous ratings. There are ten forms of the ICC, and they can give quite different values, so the most important part of the write-up is saying which one you used.
What you need before you write a single word
- The ICC form, defined by three choices:
- Model: one-way random, two-way random (raters are a sample of possible raters), or two-way mixed (these raters only).
- Type: absolute agreement or consistency.
- Unit: single rater or the average of the raters.
- The ICC value and its 95% confidence interval.
- Optionally, the F test with its df and p.
- The number of targets and raters.
The APA 7 format template
Inter-rater reliability was assessed with a [two-way random-effects], [absolute-agreement], [single-rater] intraclass correlation (ICC[2,1]). Reliability was [good], ICC = .XX, 95% CI [.XX, .XX], F(df1, df2) = X.XX, p = .XXX.
All 53 templates, ready to paste with the italics set: the Rescue Vault, $27 →
Formatting rules reviewers actually check:
- Name the form; "ICC = .88" alone is not interpretable.
- Report the 95% CI; the usual reliability labels should be read from the CI, not the point estimate.
- No leading zero on the ICC or p.
A worked example
Three raters scored 20 essays on a 100-point scale. A two-way random-effects, absolute-agreement ICC gives a single-rater ICC(2,1) of .88, 95% CI [.76, .95], F(19, 38) = 26.92, p < .001, and an average-rater ICC(2,3) of .96, 95% CI [.90, .98]. Written up in APA 7:
Three raters independently scored all 20 essays. Inter-rater reliability, assessed with a two-way random-effects, absolute-agreement intraclass correlation, was good for a single rater, ICC(2,1) = .88, 95% CI [.76, .95], F(19, 38) = 26.92, p < .001, and excellent for the mean of the three raters, ICC(2,3) = .96, 95% CI [.90, .98].
How to describe the value
Koo and Li (2016) suggest: below .50 poor, .50 to .75 moderate, .75 to .90 good, above .90 excellent, judged on the 95% CI.
Mistakes reviewers catch
- Not naming the ICC form. Reviewers cannot judge the value without it.
- Reporting the average-measures ICC when one rater will score in practice. If future scores come from a single rater, the single-measures ICC is the relevant one.
- Using Pearson's r for agreement. Two raters can correlate perfectly while one scores ten points higher every time; absolute-agreement ICC catches that.
Let KyroStat do the write-up for you
KyroStat computes the ICC forms for your rating design, says which one fits, and writes the reliability sentence with its confidence interval.
Frequently asked questions
ICC or Cohen's kappa? ICC for continuous or interval ratings; Cohen's kappa for categorical codes.
Which ICC form should I choose? For a sample of raters whose ratings should be interchangeable, two-way random-effects with absolute agreement is the usual choice. Say why in your Method.
Is the ICC the same as Cronbach's alpha? The average-measures consistency ICC equals α, but they answer different questions. See how to report Cronbach's alpha.
References
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012