What can a model actually see?
Isolated signs, from video, in good conditions. That is the boundary of what we can currently demonstrate — and the gap between that and a conversation is enormous.
Octopus Foundation · Research initiative
Auslan recognition, researched in the open.
A non-commercial initiative studying whether machine learning can support access to Australian Sign Language. We publish the method, the dataset terms, the numbers — and the experiments that failed.
Exploring
Where the work stands
One model recognises 3,215 Auslan glosses from video, running entirely in the browser. It is a research artefact — accurate enough to study, nowhere near reliable enough to depend on.
- 87.9%
- Top-1 on held-out studio validation
- 85.8%
- Top-1 on the in-the-wild test set
- 3,215
- Glosses the model can name
The question
Interpreters are scarce. That is a real problem — and an easy one to answer badly.
Deaf and hard-of-hearing Australians navigate services built around spoken and written English, and qualified Auslan interpreters are in short supply. The tempting response is to automate the gap away. We think the honest question is narrower.
What does it get wrong?
We evaluate across four recording conditions rather than reporting one flattering number, because the difference between studio and in-the-wild is where the honest story is.
What should it never do?
Recognition is not interpretation. A system that guesses confidently in a medical or legal setting is worse than no system at all, so refusing to answer is a first-class outcome.
Approach
Recognition proposes. Interpretation disposes.
The recognition model never gets the last word. It emits evidence — candidate glosses, confidences, skeletal and facial features — and a separate layer decides what, if anything, that evidence supports. The evidence is never rewritten to fit the answer.
- On-device landmark extractionMediaPipe hand, pose and face landmarkers run in the browser. Video frames are processed and discarded — nothing is uploaded, nothing is recorded.
- 55-point skeleton, normalisedBoth hands and upper body, scaled by shoulder width so the model sees the sign rather than the signer’s distance from the camera.
- Recognition over 3,215 glossesA single model trained on two independent keypoint sources, which is what makes it hold up when the extractor changes.
- Gating, then interpretationMotion and handshape activity gates suppress non-signing. Only what survives reaches the language layer — which is told to flag doubt rather than smooth it over.
Publishing negatives
The failures are part of the record.
Four things we expected to work did not. Each cost real compute, and each is the kind of result that quietly goes unpublished — which is exactly why it is here.
Face landmarks in the classifier
Facial grammar carries real meaning in Auslan, so adding face points looked obvious. Accuracy dropped. The cues stayed in the system — as evidence, not as classifier input.
Depth for handshapes
The dataset is RGBD, so depth was free to try. Naively added, it made handshape classes less separable rather than more.
A frozen RGB vision branch
Level with pose alone, inside noise — and no better on the handshape probe it was meant to help with. Dropped.
BSL-to-Auslan transfer
The shared BANZSL lineage made cross-language transfer look like free data. Every variant came back null.
Position
Auslan belongs to the Deaf community, not to a model.
Auslan is a full natural language with its own grammar, spatial structure and culture. It is not English rendered on the hands, and a system that treats it as a gesture-to-word lookup has already misunderstood it.
This work does not treat deafness as something to be fixed. It does not aim to replace interpreters — a qualified human interpreter carries responsibility no current system can hold. And it cannot be validated without Deaf leadership, which is the part that cannot be substituted with more compute.
Nothing here is a product
The demo is a research artefact. It is not validated for clinical, legal, educational or safety-critical use, and it should not be relied on in any situation where being wrong matters.