What a 97% Neutral-or-Better Score Can Actually Prove
A 97% neutral-or-better sentiment score only matters when you know the measurement window and what "neutral-or-better" actually detects. In one transit and paratransit deployment, sentiment was scored on 87% of inbound calls over a single month, and 97 of those scored calls ended in a neutral or positive state. That is a real, labeled measurement, not proof the AI understood how anyone felt. It is evidence that a pattern-detection signal, applied broadly and reported honestly, found few conversations trending negative.
What does a 97% neutral-or-better score actually measure?
It measures the output of a sentiment-classification model applied consistently across a stated batch of calls in a specific window. In this case, the batch covers one transit and paratransit deployment, one month, and 87% of inbound conversations, reported as measured rather than projected. The 97% represents the share of those scored calls that ended neutral or positive. It is not a satisfaction survey. It is not a claim about what riders subjectively felt. Sentiment models detect patterns in language and tone correlated with emotional states, but they do not verify the state itself.
That distinction matters because the peer-reviewed evidence on emotion inference is unambiguous. How people communicate emotion "varies substantially across cultures, situations, and even across people within a single situation" (Barrett et al., 2019, a review of facial-expression research; the review addresses facial movements specifically, and vocal inference is less settled still). No model, whether human or machine, reliably decodes a specific internal feeling from an external signal alone (Barrett et al., 2019, Psychological Science in the Public Interest). The honest claim is therefore narrower than "97% of riders were happy." It is "97% of scored conversations showed no signal of distress, on a channel scored broadly rather than sampled."
Why does coverage matter more than the headline number?
The 97% is meaningless without the 87%. Traditional call-center quality assurance reviews a small fraction of calls, commonly 1 to 2%, usually selected because someone complained or a supervisor pulled a random sample. A score built on that sample describes the sample, not the business. Scoring 87% of a month's inbound volume captures the routine, unremarkable calls nobody ever flags for manual review. That is exactly where quiet dissatisfaction hides. A 99% satisfaction figure drawn from 2% of calls is a much weaker claim than a 97% figure drawn from 87% of calls, even though the second number is lower. A related failure mode is covered in the wrong question about caring.
| Metric | What it tells you | What it doesn't tell you |
|---|---|---|
| 87% of calls scored | The result covers most real volume, not a curated sample | Nothing about quality by itself |
| 97% neutral-or-better | Few scored calls showed a negative signal | What any individual caller actually felt |
| The remaining 3% | Where the operationally fixable problems concentrate | Whether every one of those calls got resolved |
What is the 3% actually for?
The useful part of this metric was never the 97%. The value lies in the three in a hundred that went the other way. On a transit line, negative sentiment clusters around predictable, fixable events: a ride that failed to arrive when expected, a rider who could not get a straight answer about eligibility, or a caller who needed a person and did not reach one fast enough. In paratransit specifically, most riders in that situation do not escalate. They just wait, or they refuse to call again. A sentiment signal that surfaces that 3% while it is still fixable does real work. A sentiment signal that only shows up in a monthly aggregate arrives too late, well after the moment to intervene has passed.
What should this number never be used to claim?
Vendors should never use this metric to claim the AI knew how any individual rider felt, that the system is "empathetic," or that a 97% score means near-zero dissatisfaction across the board. Rosalind Picard's affective computing group at MIT originated this field in 1995 and has always framed the discipline around detecting and responding to emotional signals, not replicating human feeling (MIT Media Lab, Affective Computing Group). Regulators are drawing the same line. The EU AI Act's Article 5 restricts treating inferred emotion as settled fact in sensitive contexts, precisely because that inference carries real risk when it is wrong (EU AI Act, Article 5). A sentiment score is a triage tool. Its value lies in what it triggers: faster attention to the calls that need it, rather than the size of the number itself.
Before trusting any vendor's sentiment figure, demand the coverage rate and the measurement window first. Then ask what happens operationally to the calls that do not score neutral-or-better. See how that question connects to sentiment as early warning, and review AiPRL's analytics approach to reporting measured, labeled results rather than projected ones.