How to Report Cohen's Kappa in APA 7 (With Example and Table)

Cohen's kappa (κ) measures how well two raters agree when they sort the same items into categories, after removing the agreement you would expect by chance. This guide shows what kappa corrects for, how to compute its confidence interval, the APA 7 format, and a table for several coded variables.

What kappa corrects for

Raters assigning categories at random still agree some of the time, and if both put most items in one common category, chance agreement is high. Percent agreement ignores this; kappa does not (Cohen, 1960):

κ = (po − pe) / (1 − pe)

where po is the observed proportion of agreement and pe is the agreement expected by chance from each rater's category totals. κ = 1 means perfect agreement, κ = 0 means agreement no better than chance, and negative values mean worse than chance.

What you need before you write a single word

1. The number of items coded and the number of raters (Cohen's kappa is for exactly two; for three or more raters, Fleiss' kappa is the usual choice).
2. Whether the raters coded independently, and whether they coded all items or a double-coded subsample.
3. Percent agreement, so readers see the raw picture.
4. κ and its 95% confidence interval.
5. Optionally, the z test of κ against zero. It is rarely informative (agreement better than chance is a low bar), so the interval matters more.

The APA 7 format template

Two raters independently coded all [N] [items] into [k] categories. They agreed on [x] of [N] items ([percent]%), Cohen's κ = .XX, 95% CI [.XX, .XX].

boltAll 53 templates, ready to paste with the italics set: the Rescue Vault, $27 →

Formatting rules:

• κ is not italicized. APA 7 sets Greek letters in upright type; italicize Latin symbols such as z, p and SE.
• No leading zero on κ or its confidence limits: kappa cannot exceed 1, so write κ = .70, not 0.70.
• Confidence intervals go in square brackets after the statistic, with the level stated: 95% CI [.54, .85].
• Report p to two or three decimals, or p < .001.

A worked example

Two raters independently coded 60 fictional open-ended survey responses about a campus service into one of three themes: cost, access or quality. Rater A is in the rows, Rater B in the columns:

Observed agreement. The diagonal holds the items both raters coded the same: (20 + 15 + 13) / 60 = 48 / 60 = .80.

Expected agreement. Multiply each category's row and column totals, add them up, and divide by N²: (23 × 24 + 20 × 20 + 17 × 16) / 3600 = 1224 / 3600 = .34.

Kappa. κ = (.80 − .34) / (1 − .34) = .46 / .66 = .70.

Standard error and confidence interval. Using Cohen's (1960) large-sample standard error, SE = √[po(1 − po) / (N(1 − pe)²)] = √[.80 × .20 / (60 × .66²)] = 0.078. The 95% CI is κ ± 1.96 × SE = [.54, .85]. Some programs use a refined version of this formula, so their standard error can differ slightly; for these data it rounds to 0.078 either way.

Test against zero. The z test uses a different standard error, the one that assumes κ = 0: SE₀ = √[pe / (N(1 − pe))] = 0.093, so z = .697 / 0.093 = 7.52, p < .001. (All values were computed from unrounded intermediates.)

Written up in APA 7:

Two trained raters independently coded all 60 responses into one of three themes (cost, access, quality). They agreed on 48 of the 60 responses (80.0%), and chance-corrected agreement was substantial, Cohen's κ = .70, 95% CI [.54, .85], z = 7.52, p < .001.

Reporting several coded variables in an APA table

With several coded variables, use a table. Suppose the raters also coded two yes/no variables. For "mentions a specific service", both said yes on 22 responses, both said no on 31, A said yes and B no on 4, and A no and B yes on 3. For "expresses frustration", the counts were 9 both yes, 40 both no, 6 A only and 5 B only.

Table 1

Interrater Agreement for the Coded Variables

Note. N = 60 responses, each coded independently by two raters. κ = Cohen's kappa. Confidence intervals use the large-sample standard error of Cohen (1960).

Look at the last row: 81.7% agreement, more than for theme, yet κ is only .50. Most responses expressed no frustration, so chance agreement was already 63.3%, and kappa only credits agreement beyond that. Percent agreement alone would have hidden a weak variable.

Interpreting the value: labels with caution

Landis and Koch (1977) proposed labels that are still widely quoted: below 0 poor, .00 to .20 slight, .21 to .40 fair, .41 to .60 moderate, .61 to .80 substantial, and .81 to 1.00 almost perfect. They described these divisions as arbitrary.

McHugh (2012) argues that these labels are too generous for research where coding errors matter, such as health data. On her scale, .60 to .79 is moderate, .80 to .90 strong, and above .90 almost perfect, and she treats values below .60 as inadequate agreement. Under Landis and Koch, the theme variable (κ = .70) is "substantial"; under McHugh, it is "moderate". Name the scale you used, and let the confidence interval speak: a lower limit of .54 is a warning even if the point estimate looks fine.

Ordinal categories: weighted kappa

Cohen's kappa treats every disagreement as equally wrong. With ordered categories (mild, moderate, severe), mild versus moderate is a smaller miss than mild versus severe. Weighted kappa gives partial credit for near misses, using linear or quadratic weights. Say which weights you used, because they change the value, and report it the same way: weighted κ = .XX, 95% CI [.XX, .XX].

Mistakes reviewers catch

• Percent agreement only. It ignores chance and can look high for a variable the raters barely agree on, as the frustration row shows.
• κ without a confidence interval. With 60 items, the interval for κ = .70 runs from .54 to .85. That width is information.
• No detail about the coding. Readers need the number of items, the number of raters, whether coding was independent, and whether disagreements were then resolved by discussion.
• Leaning on the p value. A significant z only says agreement beat chance, which almost any trained pair of raters manages.
• Unweighted kappa on ordered categories. Use weighted kappa and name the weights.
• A leading zero or an italic κ. Write κ = .70.
• Cohen's kappa for three or more raters. It is defined for two.

Kappa uses a contingency table, like the chi-square test, but asks whether two classifications match, not whether they are associated. For other effect sizes and their formats, see the effect size guide.

Let KyroStat do the write-up for you

Formatting κ with its interval is fiddly work. KyroStat runs the analysis on your data and hands you the finished APA 7 sentences, a table and the underlying Python or R code. Upload your spreadsheet, and the report is done in seconds.

Frequently asked questions

Is Cohen's kappa italicized in APA? No. κ is a Greek letter, and APA 7 does not italicize Greek letters.

What is a good Cohen's kappa? It depends on the stakes. Landis and Koch (1977) call .61 to .80 substantial; McHugh (2012) treats anything below .60 as inadequate. Say which benchmark you use, and why it suits your data.

Should I report percent agreement as well as kappa? Yes. Together they show whether high agreement is real or an artefact of one common category.

How many items should be double-coded? There is no single rule. Report how many and how they were chosen; fewer items mean a wider interval around κ.

Which related guides might I need? Often these: intraclass correlation (ICC), Cronbach's alpha and McNemar's test.

References

American Psychological Association. (2020). Publication manual of the American Psychological Association (7th ed.).

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37 to 46.

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159 to 174.

McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276 to 282.