Statistics Claim Check

Appendix F companion • Read dashboard and AI claims before you communicate or act

Companion to Appendix F: Reading Statistical Claims in Defensible Analytics. Use this page when a dashboard, AI visual, or chat answer makes a claim about an average, a group difference, a relationship, or a driver.

This is not a formal-inference course. It does not teach hypothesis testing, regression fitting, p-value calculation, forecasting, or model selection. It helps you decide whether a claim is ready to communicate, needs more checking, or needs qualified method review before a consequential decision.

Start with the claim

Before accepting a number, name the boundaries that make it meaningful.

What is the unit?

A ticket, customer, encounter, employee, month, transaction, or something else?

What population?

All records, a named segment, or only records that survived a filter?

What denominator?

Per ticket, per customer, per dollar, per month, or per eligible person?

What comparison?

Against plan, a prior period, another group, or an expected range?

What grain?

One total, a time series, a region, a product line, or a subgroup?

What decision?

Is this for exploration, communication, recommendation, or an operational decision?

Choose the next move

Decision state When it fits Next move
Communicate a bounded description Unit, population, denominator, comparison, and material limitation are clear. State what the data shows and name the boundary.
Investigate before acting An outlier, subgroup, alternate filter, or plausible confounder could change the story. Check the distribution, a finer grain, or a focused comparison.
Seek qualified method review A consequential decision depends on an interval, significance claim, ANOVA, or model output. Review the method, assumptions, and decision consequence with a qualified analyst.

Read, do not overclaim

Average

An average is a starting point. Check the distribution, outliers, and subgroups before treating it as the whole experience.

Weighted average

Weights change the business question. Ask what they represent and who chose them.

Group difference

Name the direction, baseline, group sizes, and practical consequence before using it in a recommendation.

Association

Association is not cause. Check a scatterplot, subgroup patterns, and plausible confounders.

AI-generated driver

Read a ranked driver as a hypothesis for human review, not a decision by itself.

Significance label

A threshold is not a business decision rule or a measure of effect size. Ask whether the decision changes under a less favorable estimate.

Practice workbook

📊

Statistical Claim Workbook

Synthetic data Visible formulas Claim checks

Use the synthetic support-ticket and reimbursement examples to inspect routine formulas for descriptive summaries, weighted averages, group differences, and correlation. Model and ANOVA sheets provide labels to read, not calculations to reproduce.

Download the workbook (.xlsx)

Open the workbook in Excel, Excel for the web, or Google Sheets. The formulas are visible so you can inspect the claim, then return here to choose the next check.

Try this claim check

A support dashboard says: “Tier 2 resolution time rose 34%, so staffing is the cause.”

  1. Identify the unit, population, denominator, comparison, and grain.
  2. Open the workbook’s Kenji Raw Data and Descriptive Claims sheets. Check whether the average conceals a small number of incident-heavy months.
  3. Ask what else changed: ticket mix, identity-release timing, incident severity, or staffing.
  4. Rewrite the claim as a bounded description and name the next check before recommending any staffing action.

Go deeper

When a decision requires formal study beyond Appendix F, use an authoritative resource rather than extending a dashboard calculation beyond its evidence.

Resource Printed URL Purpose
OpenIntro Statistics https://www.openintro.org/book/os/ Free textbook, data, and exercises for formal study.
Penn State STAT 200 https://online.stat.psu.edu/stat200/ Open notes for statistical methods beyond this companion’s scope.
NIST/SEMATECH e-Handbook https://www.itl.nist.gov/div898/handbook/ Public reference for exploration, measurement, modeling, monitoring, and comparison.
Practical Statistics for Data Scientists https://www.oreilly.com/library/view/practical-statistics-for/9781492072935/ Applied bridge for analysts who need more depth on sampling, significance, regression, and misuse.

Related book resources