Data Analysis Interpreter

Analysis Plan: Flexible Remote-Work Policy and Employee Burnout at Northfield Analytics

I've built the plan from what you gave me. A few answers could change specific steps, so I've put targeted questions first. The plan below states which choice to make depending on your answer. I have not invented any data values or results.

Clarifying Questions

1. Do any employees appear in both waves? With about 450 employees and one year between surveys, most staff would normally still be employed in 2026. You said the cohorts are different people. Is that because of the survey sampling design, or because of anonymous IDs you can't link? If some people are in both waves and you can identify them, the observations are not independent, and you need a mixed model or to drop the overlap.
2. Do you have team or manager IDs? Employees on the same team share a manager and workload, so their burnout scores are correlated. If you have team IDs, use cluster-robust standard errors by team. If not, the standard errors will be somewhat too small.
3. Remote days range. Your variables list says 0–5, but both policies imply 2–5. Are there employees with 0–1 remote days (exempt roles, on-site staff)? In the pre-policy wave, is everyone exactly 2?
4. Response rate per wave. 142 and 158 out of about 450 is roughly a third. Was the survey sent to everyone each time, or to a sample?
5. What happened between March 2025 and March 2026? Examples: layoffs, reorganizations, the product-launch crunch, leadership changes. And which wave did the crunch fall in?
6. Is there a pre-agreed threshold for a meaningful burnout reduction (for example, 5 index points)? If not, I propose one below.
7. One-sided or two-sided test? Your question is directional ("lower"). I recommend two-sided, because a flexible policy could plausibly increase burnout through isolation or always-on work.

Recommended Approach

The core design compares two independent cross-sectional cohorts with a continuous outcome and pre-specified adjustment for tenure and team size. That calls for ANCOVA, run as ordinary least squares (OLS) linear regression:

• burnout is regressed on policy group, tenure, and log-transformed team size;
• standard errors are heteroskedasticity-robust (HC3), or cluster-robust by team if team IDs exist.

The key estimate is the adjusted mean difference in burnout points between cohorts, reported with a 95% CI and a standardized effect size.

With n = 300, OLS inference holds up under moderate non-normality, so a rank-based test is not needed as the primary analysis. The known issues are handled as pre-specified sensitivity analyses rather than ad hoc deletions:

• the extreme burnout scores;
• the team of 40;
• the high-overtime cases.

Three sensitivity analyses cover them:

• a robust MM-regression, which down-weights influential points instead of deleting them;
• a median (quantile) regression;
• a model that adds department.

Two design points matter more than the choice of test:

• Policy is completely confounded with calendar year. This design can show association, not a causal policy effect.
• Remote days, overtime, and manager support are probably affected by the policy itself. Adding them to the primary model would remove part of the effect you are trying to measure. They belong in secondary, explanatory models.

1. Hypotheses, Model, and Assumptions

Formal hypotheses

Primary model:

Burnoutᵢ = β₀ + β₁·Postᵢ + β₂·Tenureᵢ + β₃·log(TeamSizeᵢ) + εᵢ

• H₀: β₁ = 0. After adjusting for tenure and team size, mean burnout does not differ between the post-policy and pre-policy cohorts.
• H₁ (recommended, two-sided): β₁ ≠ 0.
• H₁ (your directional version): β₁ < 0. Use this only if it was pre-registered or agreed before looking at the data.

Why this model

Power context: with 142/158 per group, α = .05 two-sided and 80% power, the study can detect an unadjusted standardized difference of about d ≈ 0.32. Covariates that explain outcome variance improve this slightly. Smaller true effects may produce a non-significant result, and that would not be evidence of no effect.

Assumptions and how to check them

2. Descriptive Summary

Recommended statistics by variable, always split by cohort

Balance table. Add a column of standardized mean differences between cohorts. This is more useful than p-values for judging comparability.

Distribution concerns to flag

• Burnout. Look for a ceiling effect (the 98–100 cluster), and compute skewness and kurtosis per group. As rough guides, |skew| > 1 or excess kurtosis > 2 point toward the robust and quantile sensitivity models.
• Team size. Right skew with a single extreme value. Its leverage is high even though its influence on β₁ may be small; check DFBETA. The log transform handles most of this.
• Overtime. Right skew and 2 extreme values. These only matter for secondary models.
• Bivariate associations. Use Spearman's ρ for the ordinal and skewed variables. Expected correlations to check:
  • tenure with age (collinearity);
  • overtime with burnout;
  • manager support with burnout (likely negative);
  • team size with department (the 40-person pod is in Support, so team size and department are partly confounded).
• Remote days vs. policy. These are almost collinear by design, because pre-policy is fixed at 2. You cannot estimate a separate remote-days effect in the same model as policy using pre-policy data.

Step-by-Step Plan

1. Settle the design questions above: overlap between waves, team IDs, and the remote-days range.
2. Pre-specify the primary model, the sensitivity models, a two-sided α of .05, and a practical-significance threshold before looking at group differences. Write it down, even informally.
3. Check measurement quality. Compute ω or α for the burnout index in each wave. Confirm that personal-accomplishment items were reverse-scored before being combined. If item data exist, test measurement invariance across waves.
4. Produce the descriptives and balance table by cohort (Section 2), including missingness per variable.
5. Fit the primary model: Burnout ~ Post + Tenure + log(TeamSize), with HC3 SEs, or cluster-robust by team. No missing data is involved, because the outcome, policy, tenure, and team size are complete.
6. Run diagnostics: residual plots, Q–Q plot, Cook's D, DFBETA for Post, VIFs, and the slope-homogeneity interactions.
7. Run pre-specified sensitivity analyses and present them together in a forest plot:
   • a. robust MM-regression (all data retained);
   • b. median (quantile) regression, which estimates the median difference;
   • c. team size winsorized at the 99th percentile or modeled with a spline, instead of logged;
   • d. adding department (and age) as covariates, to account for changes in workforce composition;
   • e. entropy balancing or propensity-score weighting on tenure, team size, department, and age;
   • f. cluster-robust SEs by team, if not already primary.
8. Run secondary (explanatory) models, labeled as such. Add overtime and manager support to see whether the policy association weakens. Frame this as exploring pathways, not "controlling for confounders."
   • For overtime (4% missing), use multiple imputation (m ≥ 20, including burnout and policy in the imputation model).
   • Leave gender out of the primary model. If you include it, keep "Prefer not to say" as a category and multiply-impute or add a missing indicator only for sensitivity.
9. Run a within-post-cohort exploratory analysis of burnout vs. chosen remote days (2–5), as a dose-response pattern. Caveat: people chose their days, so self-selection is strong.
10. Run secondary outcomes. Repeat the primary model for each MBI subscale (exhaustion, depersonalization, accomplishment). The policy may affect exhaustion but not accomplishment, and the combined index can hide this. Apply a Holm correction across the three.
11. Report the adjusted difference with its 95% CI, the standardized effect, the sensitivity forest plot, and a plain-language summary with explicit causal caveats.

A minimal R sketch for step 5:

library(sandwich); library(lmtest)
m1 <- lm(burnout ~ post + tenure + log(team_size), data = d)
coeftest(m1, vcov = vcovHC(m1, type = "HC3"))         # or vcovCL(m1, cluster = ~team_id)
coefci(m1, vcov. = vcovHC(m1, type = "HC3"))
# Sensitivity: robustbase::lmrob(); quantreg::rq(tau = 0.5); WeightIt/entropy balancing

3. Inferential Interpretation

Reading the three numbers together

• Adjusted mean difference (β₁) with 95% CI. This is the headline, in burnout-index points. The CI shows the range of effects compatible with the data. A CI of −8 to −2 means something very different from −12 to +0.5, even if the point estimates are similar.
• p-value. This is how surprising the observed difference would be if there were truly no difference. It says nothing about size or importance, and p > .05 does not mean "no effect."
• Effect sizes:
  • Standardized: adjusted difference ÷ pooled SD of burnout (Cohen's d, with a Hedges' g small-sample correction). Rough benchmarks are 0.2 small, 0.5 medium, 0.8 large, but in organizational research, domain anchors beat generic benchmarks.
  • Partial ω² for policy. It is less biased than partial η².
  • Optional: the difference in the proportion of employees above a high-burnout threshold, reported as a risk difference with its CI. Leaders often find this most intuitive.

Statistical vs. practical significance

The burnout index is a custom MBI-derived 0–100 score, so there is no established minimal important difference. I suggest a pre-specified practical threshold using one of these anchors:

• Distribution-based: about 0.5 SD of baseline (pre-policy) burnout, a commonly used heuristic for minimally important change in self-report scales. A more lenient option is 0.3 SD.
• Consequence-based: the difference in burnout linked to meaningful outcomes in your own HR data, such as turnover intent, sick days, or exit rates, if available.
• Stakeholder-based: the reduction HR leadership would consider worth the cost and complexity of a company-wide rollout.

Then classify the result by where the whole CI sits relative to that threshold:

Limitations and confounders that could weaken the conclusion

• Policy = year. Any company-wide or economic change between March 2025 and March 2026 is inseparable from the policy. Examples: product cycle, layoffs, AI-tool adoption, the job market, seasonal workload. This is the biggest threat.
• Survivorship and attrition bias. The most burned-out 2025 employees may have left before 2026. That would make the post cohort look better even with no policy effect. New hires are also usually in a "honeymoon" period. Check tenure balance and run a sensitivity model restricted to tenure > 1 year.
• Non-response bias. At about a third response per wave, who responds may differ by wave. For example, highly burned-out employees may skip surveys, or be more likely to respond to complain.
• Composition shifts. Changes in department mix or team structure between waves. Handled partly by sensitivity model 7d.
• Mediators vs. confounders. Adjusting for overtime, manager support, or remote days would bias the total policy association (overadjustment).
• Self-selection within the post cohort. Remote-day choices reflect personal preferences, caregiving, and pre-existing burnout, so the dose-response analysis is descriptive only.
• Clustering. Ignoring team or manager nesting understates uncertainty.
• Measurement. The combined index may hide opposing subscale effects, and a single survey time point may reflect temporary states. Self-report can also be affected by demand characteristics, since employees may know the policy is under evaluation.
• Generalizability. Results come from one company and one year, and may not carry over to other departments' future conditions.

4. Visualization

1. Raincloud plot (half-violin + boxplot + jittered points) of burnout by policy group. Overlay the adjusted means with 95% CI error bars.
   • It shows the distribution shape, the 98–100 ceiling cases, and the headline comparison together.
   • Y-axis: "Burnout index (0–100, higher = more burnout)". Add "n = 142" and "n = 158" in the group labels.
2. Forest (coefficient) plot of the adjusted difference across models. Include the primary model, each sensitivity model, and the three subscales.
   • Add a vertical line at 0 and a shaded band for ±the practical threshold.
   • It shows at a glance whether the conclusion is robust, and whether it is practically meaningful rather than only significant.
3. Scatter of burnout vs. tenure, colored by cohort, with LOESS or linear fits and CI ribbons.
   • It checks slope homogeneity and the new-hire effect visually.
   • An alternative for the post cohort: mean burnout (95% CI) by chosen remote days (2–5), labeled "exploratory, self-selected."
   • Manager support is best shown as a diverging stacked bar chart by cohort.

Accessibility guidance:

• Use the colorblind-safe Okabe–Ito palette, e.g., pre-policy #E69F00 (orange) and post-policy #0072B2 (blue).
• Don't rely on color alone: add redundant encoding (different point shapes or line types).
• Label directly on the plot rather than only in a legend.
• Use at least 11–12 pt text, and keep a light-gray grid behind the data.
• Include axis units, group n, a caption stating what is adjusted and that error bars are 95% CIs, and alt text for reports.

Plain-Language Interpretation

Here is how to explain the eventual result to non-statisticians. The bracketed values will come from your analysis.

"We compared burnout between employees surveyed in 2025, under the old fixed schedule, and a different group surveyed in 2026, under the flexible policy. We adjusted for how long people had been at the company and the size of their teams.

After those adjustments, the 2026 group's burnout scores were on average [X points] [lower/higher] on the 0–100 scale. Our best range for the true difference is [L to U] points.

We decided in advance that a change of about [threshold] points would be meaningful for employees. [The result clears / may not clear / clearly does not reach] that bar. It held up [consistently / inconsistently] when we handled the extreme cases and the unusually large team in different ways.

This shows the flexible policy is associated with [lower/similar/higher] burnout. It does not prove the policy caused the difference. Other things changed over the year too, and the most burned-out people from 2025 may have already left. It is suggestive evidence to weigh alongside other information, not a definitive verdict."

How to handle the rollout decision:

• If the result is favorable, a practical next step is to build in a stronger evaluation of the extension. Options include a staggered rollout by department, or re-surveying the same employees over time. Either would allow a before/after comparison on the same people, or a comparison against a group not yet on the policy.
• If the result is inconclusive, the honest message is "we can't tell yet," not "the policy doesn't work."

Assumptions & Caveats Checklist

Design

• Confirmed no employee overlap between waves, or handled it
• Team or manager clustering addressed (cluster-robust SEs or a stated limitation)
• Policy–year confounding stated explicitly; conclusions framed as associational
• Attrition and new-hire bias assessed (tenure balance; tenure > 1 yr sensitivity)
• Response rate per wave reported; non-response bias discussed
• Major company events between waves documented

Model

• Primary model and practical threshold pre-specified before seeing results
• Two-sided test (or a justified, pre-declared one-sided test)
• Only pre-treatment covariates in the primary model; remote days, overtime, and manager support reserved for secondary models
• Linearity checked; team size logged or splined
• Slope homogeneity (Post × covariate) checked
• HC3 or cluster-robust SEs used; residual plots reviewed
• Influence diagnostics run (Cook's D, DFBETA for Post)
• VIFs acceptable
• Covariate balance (SMDs) reported, including department mix

Data issues

• Three 98–100 burnout scores retained; robust and quantile sensitivity reported
• Team of 40 handled via log/spline/winsorizing sensitivity, not silent deletion
• Two overtime values of 45+ h checked for influence in secondary models
• Overtime (4% missing) multiply imputed in secondary models
• Gender: "Prefer not to say" kept distinct from missing; not required in the primary model

Measurement

• Reverse-scoring of personal accomplishment verified
• Reliability (ω/α) reported per wave
• Subscale analyses run with multiplicity correction

Reporting

• Adjusted difference + 95% CI + standardized effect size + p-value reported together
• CI interpreted against the practical threshold
• Sensitivity results shown side by side (forest plot)
• Non-significant results described as inconclusive, not "no effect," unless equivalence is shown
• Exploratory analyses (remote-days dose-response, mediators) clearly labeled

If you answer the clarifying questions, especially team IDs, wave overlap, and the practical threshold, I can lock in the exact primary specification and draft the analysis code and report template.