Step 1: Name your question
Every test answers one of three questions: Is there a difference between groups or conditions? Is there a relationship between variables? Can one variable predict another? Write your question in one sentence before opening any software.
Step 2: Identify the type of each variable
- Categorical (nominal): categories with no order, such as gender or region.
- Ordinal: ordered categories, such as a Likert item or education level.
- Continuous (interval or ratio): numbers where distances matter, such as age, income or a scale score.
The outcome variable (what you measure) and the grouping or predictor variable decide most of the choice.
Step 3: Count the groups and check whether samples are related
Are you comparing two groups or three or more? Are the observations independent (different people in each group) or paired (the same people measured twice, or matched pairs)? Paired data need paired tests.
Step 4: Use this decision table
- Compare a continuous outcome between two independent groups: independent-samples t-test, or Mann-Whitney U if the data are strongly skewed or ordinal.
- Compare a continuous outcome before and after (paired): paired t-test, or Wilcoxon signed-rank test.
- Compare a continuous outcome across three or more groups: one-way ANOVA, or Kruskal-Wallis if assumptions fail.
- Association between two categorical variables: chi-square test of independence, or Fisher’s exact test when expected counts are small.
- Relationship between two continuous variables: Pearson correlation, or Spearman’s rho for ordinal or non-normal data.
- Predict a continuous outcome from one or more predictors: linear regression.
- Predict a yes/no outcome: logistic regression.
Step 5: Check the assumptions
Parametric tests such as the t-test and ANOVA assume roughly normal residuals, similar variances between groups and independent observations. Look at histograms and plots first. Formal tests such as Shapiro-Wilk help but can flag trivial departures in large samples, so judge them together with the plots. With reasonably large and balanced groups, the t-test and ANOVA are fairly robust. When assumptions clearly fail, use the non-parametric alternative.
Step 6: Report more than the p-value
A p-value says whether a result is unlikely under the null hypothesis. It does not say how big or important the effect is. Report an effect size (such as Cohen’s d, eta-squared or an odds ratio) and a confidence interval, and state the exact test and sample size. For help with the maths, try our free p-value calculator.
Common mistakes to avoid
- Running many tests and reporting only the significant ones. Adjust for multiple comparisons or state the tests in advance.
- Using several t-tests instead of one ANOVA for three or more groups.
- Treating ordinal data as if it were continuous without justification.
- Ignoring paired structure, for example repeated measures on the same people.
Non-parametric alternatives at a glance
When the assumptions of a parametric test fail, there is usually a rank-based alternative:
- Independent-samples t-test → Mann-Whitney U
- Paired t-test → Wilcoxon signed-rank
- One-way ANOVA → Kruskal-Wallis
- Repeated-measures ANOVA → Friedman test
- Pearson correlation → Spearman’s rho
- Chi-square test → Fisher’s exact test when expected counts are small
Worked example 1: comparing two groups
A trainer compares exam scores for two independent groups of 30 learners. Group A has a mean of 72.4 (SD 8.1) and group B has a mean of 66.9 (SD 9.3). The outcome is numeric and the groups are separate people, so an independent-samples t-test fits. The test gives t(58) = 2.44 and p ≈ .018, and Cohen’s d is about 0.63, a medium effect. A reporting sentence would read: “Group A scored higher than group B (M = 72.4, SD = 8.1 vs M = 66.9, SD = 9.3), t(58) = 2.44, p = .018, d = 0.63.” You can check the numbers with our t-test calculator.
Worked example 2: two categorical variables
In a survey of 200 adults, 30 of 100 people in one group report a health problem against 15 of 100 in another. Both variables are categorical, so a chi-square test of independence applies. The expected count in every cell is above 5, which meets the usual rule of thumb. The test gives χ²(1) = 6.45, p = .011, and the effect size (phi) is about 0.18, a small to moderate association. With Yates’ continuity correction the result is χ² = 5.62, p = .018, so the conclusion is the same.
Effect sizes: how big is the difference?
A small p-value in a large sample can describe a trivial difference. Always report an effect size alongside it. Commonly quoted benchmarks, which you should treat as rough guides rather than rules, are:
- Cohen’s d: about 0.2 small, 0.5 medium, 0.8 large.
- Correlation r: about 0.1 small, 0.3 medium, 0.5 large.
- Eta-squared (ANOVA): about 0.01 small, 0.06 medium, 0.14 large.
Sample size and power
Power is the chance of detecting a real effect. A common target is 80%. To detect a medium difference (d = 0.5) between two independent groups at the 5% level, you need roughly 64 people per group. To detect a medium correlation (r = 0.3) you need roughly 85 people. Smaller effects need much larger samples. Plan this before collecting data, because a study that is too small cannot show a real effect.
Multiple comparisons
Every test carries a chance of a false positive. If you run several tests, adjust for it. The simple Bonferroni correction divides the significance level by the number of tests, so three comparisons at 0.05 use a threshold of about 0.017 each. For follow-up comparisons after ANOVA, post-hoc methods such as Tukey’s test handle this for you.
Frequently asked questions
Which test should I use for Likert scale data?
For a single Likert item, ordinal methods such as Mann-Whitney or Kruskal-Wallis are usually safer. For a multi-item scale score, t-tests and ANOVA are often used when the distribution is reasonable.
What is the difference between a t-test and ANOVA?
A t-test compares two groups. ANOVA compares three or more groups in one test, which avoids the extra false positives that come from running many t-tests.
What sample size do I need?
It depends on the effect size you want to detect, the test and the power you want, commonly 80%. A power analysis before data collection gives the answer.
What if my data are not normal?
Use a non-parametric test, transform the data, or use a method that does not assume normality. With large samples, parametric tests are often still acceptable.
Related tools and services
Want an expert to check or run your analysis?
Send us your question and data description. We reply with a clear plan and a quote.
Discuss your project on WhatsApp