When analyzing survey cross-tabs, a residual t-test is an essential statistical tool to identify values that significantly deviate from what we expect. These deviations help uncover meaningful insights that might be missed when scanning large tables. This article explains how residual t-tests work and how to interpret the results.
What is a Residual T-test?
A residual t-test measures the difference between observed and expected frequencies in a survey data cross-tabulation and evaluates whether the deviation is statistically significant.
Residual: The difference between the observed value (actual data) and the expected value (based on the overall row and column averages).
T-test: A statistical test applied to determine whether the residuals are significantly high or low compared to what random variation would suggest.
In MX8 Lab's cross-tabs:
Green bars represent significantly higher-than-expected values (positive residuals).
Red bars indicate significantly lower-than-expected values (negative residuals).
Blue bars are not statistically significant—these values fall within the expected range.
How Does it Work?
1. Calculate Expected Values:
Each cell’s expected value is computed from the row and column totals. For instance, if a platform’s general visibility rate is 60% across services, you’d expect around 60% visibility for any specific service unless other factors influence it.
2. Compare Observed vs. Expected:
The t-test checks whether the observed value differs from the expected value by more than what can be attributed to random chance. This difference is quantified as a residual.
3. Highlight Significant Results:
• If the residual exceeds a set threshold (as defined in the project or report, typically a 95% confidence level or a p-value < 0.05), the cell is highlighted:
Green for high residuals.
Red for low residuals.
4. Account for Context:
Even slight differences might be flagged as significant in rows or columns where the expected values are exceptionally stable or predictable.
Example: Ad Visibility by Video Platform
Let’s analyze this cross-tab to understand these concepts better.
Key Observations:
1. Amazon Prime Video and YouTube (Total):
YouTube's visibility score of 81% is marked green, indicating it is statistically higher than expected. This means YouTube’s visibility score outperforms the benchmark calculated from the other services.
Conversely, 42% of Amazon Prime Video is marked red, statistically lower than expected.
2. By First Video Platform:
On Instagram, YouTube’s visibility is green at 86%, significantly higher than expected for Instagram users. However, Disney+'s visibility, at 49% (blue), is not statistically different from expectations, even though it appears close in percentage to others in the same column. This indicates subtle variations in expected rates that affect significance.
On Snapchat, YouTube’s visibility is marked blue at 78%, which aligns with the expected rate for users. In contrast, Amazon Prime Video’s 14% (red) visibility is unexpectedly low, likely because Snapchat users do not favor it for video discovery.
3. Netflix on TikTok:
TikTok users show a statistically low ad visibility for Netflix (31%, red). Despite Netflix’s broad popularity, it appears TikTok users engage less with ads on Netflix compared to other platforms. This insight may help tailor marketing strategies.
Why Do Some Similar-Looking Numbers Differ?
It’s common to see two adjacent cells with close percentages, one marked as significant and the other not. This discrepancy arises because residual t-tests consider the expected value for that row or column, not just the raw percentage differences. For example:
A 43% ad visibility for Discovery+ on Facebook (blue) is not statistically significant because it aligns with Facebook’s average ad visibility expectations.
However, a slightly higher percentage, like YouTube’s 62% on the same platform (green), is significant because YouTube’s expected visibility on Facebook was lower.
This underscores the importance of understanding the statistical framework rather than relying solely on visual comparison.
Key Takeaways:
Residual t-tests add context: They show where observed data significantly deviates from expectations, helping identify where performance exceeds or falls short.
Colors simplify interpretation: Green (higher), red (lower), and blue (as expected) offer a quick visual guide to significant insights.
Subtle differences matter: Statistical significance is relative to the expected values for each row or column, not just the raw percentages.
By leveraging residual t-tests, survey analysts can move beyond surface-level observations and uncover deeper, actionable insights. For example, platforms like YouTube may prioritize advertising on Instagram based on its standout performance, while Netflix may need strategies to improve TikTok engagement.