Worked Example: Survey Data Analysis from Raw Data to Results

The scenario

A training organisation asks 60 participants to rate a course on five statements (Q1 to Q5) from 1 (strongly disagree) to 5 (strongly agree). Thirty took the course online and thirty in person. Research question: do in-person participants report higher satisfaction than online participants?

Please note: these are simulated demonstration data, built to include typical data problems. They are not from a real survey. Download the dataset (Excel) and repeat the steps yourself.

Step 1: Look for problems in the raw data

Before any analysis, check each variable for missing values and impossible values. In the raw file we found:

• 4 blank cells: Q3 for R008, Q2 for R022, Q5 for R041 and Age for R051.
• 1 impossible rating: Q4 for R014 is 55, almost certainly a typing slip for 5.
• 1 impossible age: R034 has age 250.

In Excel, filter each column or use =COUNTBLANK(D2:H61) for blanks and =COUNTIF(D2:H61,">5") for values above the scale maximum. Our cleaning checklist lists these checks in order.

Step 2: Decide and document what to do

• We cannot be sure that 55 means 5, so we treat it as missing instead of guessing. The age of 250 is also set to missing.
• We keep the original file untouched and record each change in a cleaning log: 2 values set to missing.
• For the satisfaction score we require all five items, so respondents with any missing item (R008, R014, R022, R041) are left out of that score. That leaves 56 of 60 respondents: 27 online and 29 in person.

This is a simple choice that suits 7% missing. With more missing data, look into the pattern before dropping cases. See our missing data and outlier handling page.

Step 3: Describe the sample

Age was available for 58 respondents: mean 35.3 years, SD 10.1, median 34, range 19 to 53. Group sizes were equal in the raw data (30 and 30).

Step 4: Check that the five items form a reliable scale

Cronbach’s alpha for the five items (56 complete cases) is 0.91, which indicates high internal consistency, so averaging the items into one score is reasonable. You can reproduce this with our Cronbach’s alpha calculator. An alpha this high can also mean items are very similar, so read the items before assuming the scale is good.

Step 5: Compute the satisfaction score and describe each group

The score is the mean of Q1 to Q5. In Excel, with items in D to H: =IF(COUNT(D2:H2)=5,AVERAGE(D2:H2),"").

Step 6: Choose and run the test

We compare the means of two independent groups on a score that is approximately continuous, so an independent-samples t-test fits. Because group variances and sizes differ slightly, we use Welch’s t-test, which does not assume equal variances. A normality check (Shapiro-Wilk) was fine for the online group (p = .56) but not for the in-person group (p = .003), because many scores sit near the maximum of 5. So we also run a Mann-Whitney U test as a robustness check. See how to choose a statistical test.

Both tests agree, which makes the conclusion more convincing. You can check the t-test with our free t-test tools.

Step 7: Write up the result

Step 8: State what the result does and does not show

• It shows that, in this sample, in-person participants rated the course about 0.85 points higher.
• It does not show that in-person delivery caused higher satisfaction. If people chose their format, differences between those who chose each could explain the gap.
• A large effect on simulated data does not predict what a real survey would find.
• Report the confidence interval as well as the p-value, because it shows how precise the estimate is.

What to take from this example

1. Check and document data problems before analysing.
2. Check scale reliability before combining items.
3. Pick the test to fit the data and the design, and run a robustness check when assumptions are doubtful.
4. Report effect size and confidence interval, not only p.
5. Be clear about what the design allows you to conclude.

If you want this kind of cleaning, analysis and write-up done for your own survey, see survey data analysis and Likert scale analysis.