The p value in statistics quantifies the probability of observing results as extreme as, or more extreme than, those in a sample-if the null hypothesis is true. It does not measure the probability that the null hypothesis is true, nor does it indicate effect size or practical importance.
A p value is derived from a test statistic under a known sampling distribution, such as the t-distribution or normal distribution. It reflects how likely the observed data would be under the assumption that no real effect exists. A low p value suggests the data are inconsistent with the null hypothesis, prompting researchers to reconsider it.
For example, in a clinical trial testing whether a new drug reduces blood pressure, the null hypothesis states no difference between treatment and placebo. If the p value is 0.02, there is a 2% chance of seeing such a result-or more extreme-if the drug truly has no effect. This leads many researchers to reject the null hypothesis at the conventional 0.05 threshold.
P values depend on the test statistic, sample size, and assumed distribution. For a two-tailed t-test comparing two group means, the test statistic is computed as:
t = (x̄₁ - x̄₂) / √(s²₁/n₁ + s²₂/n₂)where x̄₁ and x̄₂ are sample means, s² is the pooled variance, and n is sample size. The resulting t-value is then compared to the t-distribution with degrees of freedom equal to n₁ + n₂ − 2. The area under the curve beyond the absolute value of t gives the p value.
Modern software computes this automatically. In R, the command pt(t_stat, df, lower.tail = FALSE) * 2 returns the two-tailed p value. In Excel, the function T.DIST.2T(t_value, degrees_freedom) performs the same calculation.
When sample sizes grow large, even trivial differences can produce small p values. A study with 10,000 participants may find a p value below 0.05 for a weight change of just 0.1 kg-statistically significant but practically negligible.
There is no single "type" of p value, but it is frequently misused in four ways:
According to the American Statistical Association (ASA), "p-values do not measure the probability that the studied hypothesis is true." The ASA's 2016 statement emphasized that p values should not be used as the sole criterion for decision-making in science.
Another common error is treating p = 0.049 as meaningful and p = 0.051 as unimportant-despite both indicating similar levels of evidence. The difference between them is arbitrary and rooted in convention, not logic.
In medical research, p values help assess whether a treatment effect is likely due to chance. For instance, a 2018 study on statins found a relative risk reduction of 20% in heart attacks (p = 0.003). This low p value supported rejecting the null hypothesis that statins have no benefit.
However, the same study reported a 0.5% absolute risk reduction. While statistically significant, the clinical impact was modest. A patient considering statins would need to weigh this small benefit against potential side effects and cost.
Consider a marketing A/B test comparing two website layouts. Layout A converts at 5.2%, Layout B at 5.4%. With 10,000 visitors per group, the p value is 0.041. The result is statistically significant, but the 0.2% increase in conversion may not justify redesign costs.
This illustrates a core principle: statistical significance ≠ practical relevance. The p value tells you whether an effect is detectable, not whether it matters.
Understanding p values requires distinguishing them from related ideas:
| Term | Definition | Key Difference from p value |
|---|---|---|
| Effect size | Measure of magnitude of an observed relationship (e.g., Cohen's d, r). | Indicates how large the difference is, independent of sample size. |
| Confidence interval | Range of plausible values for a population parameter. | Shows uncertainty around an estimate; a 95% CI that excludes zero implies p < 0.05. |
| Power | Probability of detecting an effect if it exists. | Related to sample size and effect size; high power reduces false negatives. |
For example, a study with a small sample size may fail to reject the null hypothesis (high p value) even when a real effect exists-this is a Type II error. Increasing sample size improves power and reduces the chance of missing true effects.
Use p values cautiously. They are tools, not verdicts. The American Psychological Association recommends reporting effect sizes and confidence intervals alongside p values to give a fuller picture.
Some journals now discourage the term "statistically significant" altogether. Instead, they ask authors to describe the strength of evidence and avoid binary thinking (significant vs. not significant).
When interpreting a p value, remember: it assumes the null hypothesis is true. It says nothing about the alternative. A p value of 0.01 does not mean the alternative is 99% likely-it only means the data are unlikely under the null.
Finally, p values are sensitive to model assumptions. Violating assumptions-like normality or independence-can distort p values even if the sample size is large.
Always interpret the p value in light of design, effect size, and prior evidence-not as a standalone truth signal.