Results
Isolated recognition across four test conditions, continuous translation that is barely working, and four experiments that failed. Every figure is read back from the training artefacts rather than transcribed from notes.
Isolated recognition
Each row is a separately trained model. Accuracy is over the full vocabulary — a random guess would be under 0.04%.
| Model | Top-1 | Top-5 |
|---|---|---|
| Pose onlyDataset-supplied keypoints | 84.7% | 97.2% |
| MediaPipe onlyOur own extractor | 87.3% | 97.2% |
| Joint, dual-sourceBoth extractors, one model — shipped | 87.9% | 97.7% |
| Holistic extractorTested as a replacement — did not hold up | 86.9% | 97.3% |
| 79 points, with faceFace landmarks added to the classifier | 83.6% | 96.8% |
The dual-source model is evaluated against both keypoint sources and scores within a tenth of a point of itself on each; the stronger side is shown. Read the reasoning on the research page.
Robustness
A single headline accuracy hides the thing worth knowing. Each subset covers the full vocabulary at n=6,430, so these are directly comparable — and the spread between studio and in-the-wild is the real finding.
| Test set | Pose-only top-1 | Dual-source top-1 | Dual-source top-5 |
|---|---|---|---|
| STUStudio — same conditions as training | 87.8% | 90.6% | 98.0% |
| TEDTemporal disturbance — altered timing | 84.6% | 87.8% | 97.4% |
| SYNSynthetic background | 82.9% | 86.1% | 96.0% |
| ITWIn the wild — the honest case | 81.7% | 85.8% | 96.3% |
Dual-source training improves every condition, and improves the in-the-wild case by about four points — the largest gain lands where conditions are worst, which is the direction you want it to go. In-the-wild remains roughly five points below studio.
Continuous translation
Recognising isolated signs and translating connected signing are different problems. We can do the first reasonably well. The second is barely working, and we publish it that way.
A BLEU of 10 is a low score. We report it because the useful result is the relative one: initialising the translation encoder from the supervised isolated recogniser gained +1.64 BLEU over training from scratch. Supervised recognition transfers to translation — that is a lever worth knowing about, and it is not the same claim as having a translation system.
News content sat at roughly 1.6–1.9 no matter what we changed. The diagnosis was a vocabulary and data-scale wall: far less supervision per word, and most test sentences containing words the model had never seen. We marked it no-go rather than continuing to spend against it.
Negative results
Each cost real compute. Each is the kind of result that usually goes unpublished, which is precisely why it is here — if it saves another group the same spend, it did more good than a fifth accuracy point would have.
Adding 24 face points to the 55-point skeleton dropped top-1 from 84.7% to 83.6% on the same split. Facial grammar is real and it matters, but for isolated word recognition it added parameters without adding signal. Facial cues stayed in the system — they are routed to the interpretation layer as evidence, not folded into the classifier.
The dataset is RGBD, so depth was free to try. Feeding it in alongside the skeleton made handshape classes less separable, not more. The plausible reading is that per-frame depth noise swamped the millimetre-scale differences that distinguish handshapes. Multi-view is the more promising direction.
A dual-stream model with a frozen DINOv2 RGB branch scored 90.56% against 90.45% for pose alone on the same harness — inside noise. A handshape-specific probe showed the RGB branch was no better than pose at exactly the thing RGB was supposed to help with. The branch was dropped.
Auslan, BSL and NZSL form the BANZSL family with substantial vocabulary overlap, which makes cross-language transfer look attractive. Both encoder-transfer and co-training from BSL translation data produced null results on Auslan translation. A same-language control was also null, which locates the problem in the weak continuous-translation source rather than in the language gap.
Reading these numbers