Research
The recognition model proposes. A separate layer disposes. Keeping those two jobs in separate hands is the single design decision the rest of the system is built around.
Principle
In an accessibility setting, a fluent wrong answer is more dangerous than no answer. Everything below exists to make the system able to say it does not know.
The recognition model emits evidence: candidate glosses, calibrated confidences, the skeletal trajectory, and facial cues. It does not emit sentences. A separate interpretation layer reads that evidence and decides what — if anything — it supports.
The rule that makes this more than a diagram: the interpretation layer can never rewrite the evidence. It can decline to interpret, flag a segment as doubtful, or hedge. It cannot go back and change what the model saw so the story reads better. When those two disagree, the disagreement is preserved rather than resolved away.
Pipeline
The core result
This is the finding we would keep if we could keep only one.
Keypoint models have a brittle dependency that accuracy figures hide: they learn the quirks of whichever pose extractor produced their training data. Swap the extractor at deployment and accuracy can fall off a cliff — which matters here, because the dataset ships its own keypoints while the browser runs MediaPipe.
So we trained one model on both sources at once. The dataset’s official keypoints and our own MediaPipe extraction, same architecture, same weights, both streams. The joint model beat either single-source model on both validation sets at once — training on two extractors is not a compromise between them.
| Trained on | Top-1 | Top-5 |
|---|---|---|
| Dataset keypoints onlyDataset-supplied keypoints | 84.7% | 97.2% |
| MediaPipe onlyOur own extractor | 87.3% | 97.2% |
| Both sources, one modelShipped in the demo | 87.9% | 97.7% |
Validation accuracy over all 3,215 glosses. The joint model is evaluated against both keypoint sources; the figure shown is its stronger side, and the two differ by roughly a tenth of a point.
The test that matters
Training on two sources is only interesting if it generalises to a third. So we ran RTMW — a different pose estimator, never used in training — over the same footage and the same camera view, changing nothing but the extractor, and evaluated the model unchanged.
| Keypoints from | Top-1 | Top-5 | Seen in training |
|---|---|---|---|
| Dataset's official poseOne of the two training sources | 87.8% | 97.7% | Yes |
| MediaPipeThe other training source | 88.0% | 97.3% | Yes |
| RTMWNever seen in training — swapped in at evaluation | 87.0% | 97.2% | No |
Accuracy on the unseen extractor falls by 0.8 of a point. It still beats the single-source pose model measured on its own extractor (84.7%) by 2.3 points — so training across two extractors is not a compromise between them.
Two extractors disagree in uncorrelated ways. Forced to satisfy both, the model cannot lean on either one’s artefacts, and what survives is closer to the sign itself than to any particular way of measuring it. The practical consequence is that swapping the extractor stops being a redeployment risk — which matters, because the dataset ships its own keypoints while the browser runs MediaPipe.
Refusal
Continuous recognition drifts. Left alone it will keep naming signs through hand-waving, scratching and stillness. Three layers exist to stop that, and they fail independently.
Motion and handshape gates decide whether signing is happening at all, before recognition output is allowed through. Calibrated against fluent signers — an early version was tuned against developers imitating signs and rejected real signing as too fast.
Candidate glosses are checked for coherence against what came before. This can only reorder or suppress candidates the model already proposed — it never invents one.
The interpretation layer marks whole segments as doubtful when the evidence does not hang together. Doubtful segments are greyed out, not deleted — the reader sees that something was seen and not trusted.
Calibration
A classifier over 3,215 classes always returns a best guess, even when the clip contains no sign at all. So a separate head estimates the probability that the clip genuinely contains the gloss it named, and that estimate is what the gates act on.
Against random negatives it separates cleanly — 0.9993 AUC. The number that matters more is the hard case: against deliberately confusable negatives, signs that look alike, it holds 0.9744 AUC. The gap between those two is an honest picture of where the difficulty lives.
Facial grammar
Non-manual features are grammar in Auslan: raised brows can mark a question, and the face carries negation, topic and conditionals in parallel with the hands. Leaving them out of a signing system would be a linguistic error.
Adding them to the classifier, though, made it worse — 84.7% to 83.6% on the same split when face points were folded in. For isolated word recognition, facial channels added parameters without adding signal.
So facial cues take the other route. They are extracted, passed to the interpretation layer as evidence, and used there to inform things like whether an utterance reads as a question. This is the evidence-and-interpretation split doing actual work: a signal can matter for meaning without belonging in the recogniser.