How Often Are p-Values Misreported? What the Research Shows

A statistical result is only trustworthy if the numbers are internally consistent: the reported p-value should actually follow from the test statistic and its degrees of freedom. When researchers checked whether that is true across hundreds of thousands of published results, the answer was sobering. This is what the evidence says, why it happens, and how to keep it from happening in your own work.

The short answer

Across the largest study to date, using an automated checker called statcheck on more than a quarter of a million p-values:

• About half of the papers (49.6%) contained at least one p-value that did not match its own test statistic and degrees of freedom.
• Roughly one in eight papers (12.9%) contained a gross inconsistency: an error large enough that the reported significance was wrong, which can change the paper's conclusion.
• The errors were not random. Gross inconsistencies showed up more often for results reported as significant than for non-significant ones, a bias in favor of the hoped-for result.

Those figures come from Nuijten and colleagues (2016), and an earlier, independent study found the same pattern.

What the largest study found

Nuijten, Hartgerink, van Assen, Epskamp, and Wicherts (2016) built and validated an R tool, statcheck, that recomputes the p-value implied by a reported test statistic and degrees of freedom, then compares it to the p-value the authors printed. They ran it over 258,105 p-values from 30,717 articles (16,695 of which reported usable null-hypothesis significance tests) across eight major psychology journals from 1985 to 2013.

An inconsistency here does not mean fraud. It usually means a number was transcribed, rounded, or copied wrong somewhere between the analysis output and the manuscript. But when one in eight papers carries an error big enough to flip a result from significant to not (or the reverse), the aggregate effect on the literature is real.

It is not random: the errors favor significant results

The most important finding is not the raw error rate; it is the direction. Gross inconsistencies appeared more often among p-values reported as significant (1.56%) than among those reported as non-significant (0.97%). In other words, when a mistake crept in, it was more likely to push a result across the p < .05 line than away from it, the direction that happens to support the researcher's hypothesis. The authors described this as a systematic bias in favor of significant results.

An earlier study found the same thing

This was not a one-off. Bakker and Wicherts (2011) hand-checked the consistency of test statistics, degrees of freedom, and p-values in a sample of 281 articles from high- and low-impact psychology journals. They found that around 18% of reported results were incorrect, and about 15% of articles contained at least one error that, on recalculation, changed a statistical conclusion. As in the later study, the errors tended to fall in line with the researchers' expectations, and they were more common in lower-impact journals.

Two independent teams, different methods, the same conclusion: reporting errors are common, and they lean toward significance.

Why this happens

Almost none of this is misconduct. It is the predictable result of a manual pipeline:

• A statistic is copied by hand from software output into a manuscript, and a digit changes.
• A value is rounded inconsistently, so the printed p no longer matches the printed test statistic.
• A results paragraph is reused from an earlier draft and one number is not updated.
• No automated check ever recomputes the p-value from the test statistic before submission.

Every one of these is invisible to a human proofreader, because the sentence reads fine. Only recomputing the p-value from the test statistic and degrees of freedom catches it.

The fix: automated consistency checking

The encouraging part of this story is that the problem is fixable, and the fix is cheap. Because the check is purely arithmetic, a computer can do it in bulk. When Nuijten and Wicherts (2024) looked at what happened after a journal began running statcheck during peer review, they found a steep decline in reporting inconsistencies. Catching the error before publication, rather than years later, works.

The lesson generalizes: you should not rely on eyeballing your results table. Recompute every p-value from its test statistic and degrees of freedom before you submit.

How KyroStat prevents it

This is exactly why KyroStat re-derives every p-value it reports. The statistics are computed by real Python and R engines, and before anything is exported, each reported p-value is recalculated from its test statistic and degrees of freedom, the same consistency check statcheck applies, and any mismatch is flagged. The AI is never allowed to write or restate a number, so it cannot introduce a transcription error of its own.

The result is a write-up that is internally consistent by construction: the sentence, the table, and the underlying code all agree, because they are generated from the same computed values.

What this means for your own write-up

You do not need to wait for a tool to protect yourself. Whatever software you use:

• Recompute, do not retype. Pull p-values from output programmatically where you can, rather than copying digits.
• Check the direction. If a result is "just significant" (p = .04 to .05), double-check it against the test statistic; that is exactly where flips happen.
• Report the pieces that let a reader check you: the test statistic, both degrees of freedom where they apply, the exact p-value, and an effect size. Our reporting guides show the correct APA 7 format for each test, and the assumptions guide covers what to verify before you trust the p-value at all.

A clean, consistent results section is not just good manners. On the evidence above, it is one of the most common things that separates a defensible paper from one a reviewer, or a re-analysis, can pick apart.

References

• Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., and Wicherts, J. M. (2016). The prevalence of statistical reporting errors in psychology (1985 to 2013). Behavior Research Methods, 48(4), 1205 to 1226. Full text
• Bakker, M., and Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43(3), 666 to 678. Full text
• Nuijten, M. B., and Wicherts, J. M. (2024). Implementing statcheck during peer review is related to a steep decline in statistical-reporting inconsistencies. Advances in Methods and Practices in Psychological Science, 7(2). Article

Frequently asked questions

Does an inconsistent p-value mean the researchers cheated? Usually not. Most inconsistencies are honest transcription or rounding errors made while moving numbers from software into a manuscript. The concern is their frequency and their bias toward significant results, not intent.

What counts as a "gross" inconsistency? One where the reported p-value falls on the opposite side of the significance threshold from the value the test statistic actually implies, so the stated conclusion (significant or not) is wrong.

How can I check my own results? Recompute each p-value from its test statistic and degrees of freedom, rather than retyping it. Tools like statcheck do this for finished manuscripts; KyroStat does it automatically as part of every analysis it runs.

Is this only a psychology problem? The large-scale studies were done in psychology because its reporting format is easy to check automatically, but manual transcription of statistics happens in every field, so the underlying risk is not unique to psychology.