∿ExploreSims
ExploreSims / Statistics / Inference / Hypothesis Testing & p-Values
Premium experiment • evidence under H₀

STATISTICS • INFERENCE

Hypothesis Testing & p-Values

Change an observed sample mean, null mean, variability and sample size to see the z statistic, p-value and rejection decision.

● NULL DISTRIBUTIONEvidence against H₀
z statistic
Two-sided p-value
Significance α
Decision

Reasoning inspector

live model state
Standard error—
z statistic—
p-value—
Alpha—
Standardized effect—
Decision—

A p-value measures extremeness under the null

Move the observed mean relative to the null and see how standard error changes the z statistic. The shaded tails show which outcomes are at least as extreme as the observed result.

LEARN WITH THE SIMULATION

Understand Hypothesis Testing & p-Values by experimenting.

START HERE

A hypothesis test asks how surprising an observed statistic would be if a specified null model were true. This two-sided z-test simulation places the observed standardized mean on the null distribution, shades the rejection tails and updates the p-value and decision as sample size, variability or the observed mean changes.

BUILD THE IDEA

What you should understand first

  • Standardize the difference between a sample mean and a null mean using the standard error.
  • Interpret the two-sided p-value as tail probability under the null model rather than as the probability that the null is true.
  • See how larger samples can make the same raw mean difference more statistically distinguishable.
  • Separate the significance threshold alpha from the observed p-value.

TRY IT YOURSELF

How can the same mean difference become more statistically significant when sample size grows?

  1. Set a modest difference between the sample mean and null mean and record the z statistic and p-value.
  2. Increase sample size while keeping the means and sigma unchanged.
  3. Watch the standard error shrink and the observed z marker move farther into a tail.
  4. Change alpha afterward and notice that alpha changes the decision threshold but not the p-value itself.
Read the visual:

A larger sample reduces the standard error, so the same raw difference corresponds to a larger absolute z statistic and a smaller p-value. Alpha does not change the evidence produced by the data; it changes the rule used to classify that evidence.

WHY IT WORKS 01

Standardization converts distance into evidence

A raw difference of two units means little until it is compared with the amount of sampling variation expected under the null. Dividing by the standard error turns that raw difference into a z statistic. The same difference can therefore be ordinary in a noisy small sample but unusual in a precise large sample.

WHY IT WORKS 02

The decision rule is not the evidence

The p-value is determined by the observed statistic and the null model. Alpha is a decision threshold chosen before or independently of the observed p-value. Lowering alpha makes rejection harder, but it does not retroactively change where the observed statistic lies or how much tail probability it has under the null.

DO NOT MIX THESE UP

Common misconception

A p-value is not the probability that the null hypothesis is true, and failing to reject the null is not proof that it is true. Statistical significance also does not measure practical importance; a tiny effect can produce a small p-value when the sample is sufficiently large.

PRACTICE CHALLENGE

Can you make the test reject without changing the observed mean difference?

  • Choose sample and null means that initially produce a non-rejection.
  • Keep both means and sigma fixed, then increase only sample size.
  • Find approximately where the p-value crosses the selected alpha and explain what changed mathematically.
Reveal the reasoning

Increasing n reduces σ/√n, the standard error in the denominator of z. The raw mean difference is unchanged, but measured in standard-error units it becomes larger, pushing the statistic farther into the null distribution's tails and reducing the p-value until it can cross the rejection threshold.

MODEL BOUNDARIES

What this simulation assumes

Every simulation simplifies reality or a mathematical system. These are the assumptions and limits to keep in mind while interpreting the results.

Model assumptions

  • The test is a two-sided one-sample z test for a mean with population standard deviation treated as known.
  • The null sampling distribution of the standardized mean is modeled as standard normal.
  • The p-value is calculated from the absolute z statistic and compared with a learner-selected alpha.

Model limitations

  • Unknown-sigma t tests, one-sided alternatives, multiple testing, sequential analysis and power calculations are omitted.
  • A statistically significant result is not interpreted as practical importance or as the probability that the null hypothesis is false.

CONNECT IT

Where this matters

  • A/B tests and controlled experiments
  • Quality-control mean comparisons
  • Scientific hypothesis testing
  • Understanding statistical significance in published results

WORDS THAT MATTER

Key vocabulary

Null hypothesis
The reference claim used to generate the sampling distribution against which the observed statistic is compared.
p-value
The probability, under the null model, of obtaining a test statistic at least as extreme as the one observed.
Significance level
The preselected threshold alpha used to decide when a p-value is small enough to reject the null.
z statistic
The observed mean difference measured in standard-error units.