The Record of What Was Said

The word that came back yellow

In my own use of SL2T, some signs come back less reliably than others, and one of them is WHY. On repeated use, WHY has at times come back on the screen as YELLOW. This is not a benchmark result and I will not present it as one. It is one signer’s repeated observation of one system, and it proves nothing about SL2T’s overall accuracy. What it shows is the shape of a particular kind of failure, and that shape is the subject of this piece.

YELLOW is a valid English word. It is spelled correctly, it sits comfortably in a sentence, and nothing about it signals that a mistake has occurred. A hearing reader on the other end of a text message would see a slightly odd sentence and move on. A Deaf signer who reads English easily would catch it and fix it. A Deaf signer who does not read English easily, and there are many, including signers whose first and strongest language is ASL and who were educated in systems that never gave them full access to written English, has no visible reason to stop it. The sentence goes out under their name.

Google DeepMind’s own launch material documents the same pattern in its own words. Its 12 August 2026 announcement of SL2T lists the errors that remain: rare signs, rapid fingerspelling, where “prey” became “grey,” passive constructions, classifier depictions, where the word “claws” was dropped, and tense without context, where “kicked off” became “start” ^[1]. The joint impact report DeepMind wrote with its AI Sign Language Advisory Committee (AISLAC), whose co-authors include representatives of the National Association of the Deaf, RIT/NTID, DPAN and the World Federation of the Deaf, records a model that “consistently” distorted particular number sequences, “systematically transcribing the Maine area code 207 as 2007” ^[2]. Every one of those errors produces something that reads as English. That is what makes them dangerous.

This piece continues an exchange. In August I published a governance reading of SL2T, “The Coordinates of Trust” ^[4]. Jeff Panasuik answered it with “The Second Language Condition,” an architectural reading arguing that SL2T removed gloss but kept a written-English output target that decides what counts as success and what becomes the record ^[3]. He closed by asking which safeguard gives an institution something it cannot reinterpret, negotiate away or quietly ignore. I answered that question on 8 October: the safeguards an institution cannot quietly ignore are the ones that belong to the Deaf person, meaning a right to a qualified interpreter and the signed source as the record ^[5]. I am not repeating that answer here. This piece takes up a different question raised in the public discussion of Jeff’s article, one that answer left open: if English literacy is the precondition, can the precondition itself be changed by letting the signer check the machine’s interpretation in sign?

Where verification lives in SL2T today

It is worth being exact about how SL2T is designed to be checked, because the design is more careful than most of the field and its limits are stated by the developer itself.

SL2T runs as sign-to-text dictation in Gboard, anywhere a user would type, and as signed replies in Live Transcribe, on Pixel 11 at launch, from ASL into English ^[1]. The impact report names the design principle directly: “the signer remains directly in the loop as a verifier, and the model functions as an assistive draft-generation tool rather than an autonomous interpreter” ^[2]. In practice that principle has two parts. The generated English is shown to the user, who can review it and make corrections “using the standard touch keyboard” before sending it, and in Live Transcribe nothing is shown or voiced to the other person until the signer releases it.

Then the report adds a note that deserves to be quoted in full, because it is the most honest sentence in the release: “This human-in-the-loop safeguard presumes that the user possesses functional English literacy to verify the generated text and sufficient device proficiency to operate on-screen editing controls. Where these conditions are not met, translation errors may pass unnoticed” ^[2].

Read the flow as a sequence of tests the signer has to pass. The verification step is a reading test in English. The correction step is a writing test in English. The release decision is taken on the basis of both. The system translates out of ASL, and every check on that translation happens in the language it translated into. That is the point Jeff made architecturally, and it is the point the report makes procedurally when it says errors “may pass unnoticed.” Both are describing the same structure from different sides.

DeepMind and AISLAC also drew the boundary in the right place for this version. The report confines SL2T 1.0 to low-stakes uses such as messaging, search and casual one-on-one exchanges, and it explicitly prohibits clinical consultations, police interrogations and court proceedings, IEP meetings and classroom interpretation, job interviews and disciplinary hearings, and benefits determinations ^[2]. It states that the model “must not be used by third parties to evade statutory obligations under accessibility laws” and that it does not believe SL2T 1.0 satisfies reasonable accommodation obligations under the ADA or Section 504. That scoping is a governance achievement and I said so in August. The question here is what would have to be true before the scope could responsibly widen, and verification is the first item on that list.

Translation is not verification

The distinction between translation and verification is the most useful thing in Jeff’s article. A system can produce English that scores well against a reference translation while failing to preserve what the signer meant. DeepMind reports a zero-shot score of 70 BLEURT on the FLEURS-ASL benchmark, which it describes as significantly higher than any previously reported score ^[1]. BLEURT is a learned measure of how closely generated text resembles a reference text. It is a reasonable instrument for comparing models, and it cannot see the signer. It measures the distance between two pieces of English. Whether either piece of English is what a particular Deaf person meant at a particular moment is a question the metric was never built to answer, and the sign language translation literature has been saying for years that the field’s evaluation habits overstate what its numbers mean ^[11,12].

Verification asks a different question: did the meaning, the direction, the referents, the force, the uncertainty and the intent present in the signing survive the transformation? If the only evidence offered for survival comes from the same process being evaluated, survival has not been independently established. The system has been asked to certify itself, in the language it was optimized to produce.

That is why the obvious fix is weaker than it looks. If English is the problem, translate the English back into sign and let the signer check it. Translation research has tested this idea for decades under the name round-trip or back-translation, and the results are not encouraging. Harold Somers tested round-trip translation as a quality check on machine translation and found it “a poor predictor of quality,” with back-translations scoring well even when the forward translation was bad ^[9]. Dorothée Behr’s study of back-translation in survey research reached the same conclusion from the other direction: it leaves many real problems undetected and raises a substantial number of false alarms ^[10]. A second fluent output tends to inherit the first system’s assumptions, and where it does not, it introduces its own.

Two loops that look the same on screen

The idea that an avatar could change the verification problem came up in the public discussion of Jeff’s article. SignsMeta proposed animating an avatar directly from MediaPipe landmark data rather than asking a generative system to produce a plausible sign, so that the avatar shows the user what the system actually retained. I have not been able to retrieve a permanent link to that comment and I credit it here from my own reading of the thread. The proposal deserves a careful answer, because on a phone screen two very different designs would look almost identical.

In the first design, the system translates the signing into English, or into some internal representation of meaning, and then generates signing from that output. This is back-translation with a face on it. It may be a good interface, and it has one real virtue: a signer who sees the avatar sign YELLOW knows something went wrong without having to read a word of English. But the avatar has not established what the system perceived or where the error entered. If the first transformation turned WHY into YELLOW, the avatar faithfully signs YELLOW. It renders the system’s conclusion in a second form, and a generative avatar will render it fluently, because fluency is what generative systems are built to produce.

In the second design, the avatar is animated directly from the landmark data captured before translation. It is not producing a sign that matches the system’s conclusion. It is replaying the movement information the system retained on its way to that conclusion. That is a different evidentiary object, and it supports a different kind of reasoning.

The diagnostic value is easiest to see with the error that started this piece. If the replay visibly shows the WHY I intended while the English reads YELLOW, the information survived capture and the error happened later, in the model’s decoding or in the language modelling that pulled it toward a familiar English word. If the replay itself shows something that is not WHY, because the handshape collapsed, the movement was lost or the face did not register, the problem is earlier: in sensing, in landmark extraction, or in what the representation can hold.

Those two branches lead to different fixes and different accountable parties, which is exactly what an incident log, a correction right or a procurement remedy needs to know. A back-translation avatar cannot split them. A capture replay can.

There is a further reason the second design matters for SL2T in particular. DeepMind’s privacy design sends only coordinates to the server and allows “the original video to be discarded immediately” ^[1]. That is a genuine privacy protection, and signers’ faces and bodies deserve it. But it has a consequence the release does not discuss. Once the video is gone, the landmark sequence is the only surviving copy of what the signer did. The impact report even includes an illustration, its Figure 1, of what the model “sees” for a single frame ^[2]. A capture replay is that figure in motion. It is not a new artifact the developer would have to invent. It is the evidence the architecture already keeps.

Once the video is discarded, the landmark trace is the only surviving copy of what the signer did.

What the trace keeps and what it drops

A capture replay is only as good as the capture, and here the public record is unusually specific. MediaPipe Holistic, the on-device tracker SL2T uses, is a published Google component. Its current documentation describes 478 face landmarks, 33 pose landmarks and 21 landmarks for each hand, 553 in all, with pose and hand landmarks available in three-dimensional world coordinates ^[7]. Google Research’s original announcement described the same topology as “540+ keypoints,” using an earlier 468-point face model ^[8].

SL2T does not use all of that. The impact report states that the model’s input “consists of 2D landmark coordinates for 130 key points covering the face, body, and hands” ^[2]. The report does not say which 130 points are used or how they divide among the face, the body and the hands. What it does say is direct about the consequences: the face model “does not track landmarks on the tongue, which is a critical articulator in ASL”; the representation “captures no information about contents of the signer’s environment, which can be used for referencing real-world entities”; and “the lack of depth ordering means the distinction between contact and hovering can be difficult to perceive” ^[2].

None of this is a criticism of the decision to use landmarks. The report argues, reasonably, that the representation is efficient, privacy-protective and, with adequate training, rich enough for the uses it targets. The point is narrower and matters for verification. A replay reconstructed from 130 two-dimensional points can only show what those points hold. If a meaning was carried by the tongue, by a pointing gesture to a real object in the room, or by whether one hand touched the other or stopped just short of it, the replay cannot show it, and it will not show its own absence. It will reproduce the omission with perfect fidelity.

The non-manual problem sharpens this. The report’s own list of limitations says the model “sometimes misses critical Non-Manual Markers, such as facial expressions, eyebrow positioning, and head nods or shakes, which convey negation, questioning, and core grammatical affect,” and its P1-2 issue records that “lexemes conveyed purely with NMMs often fail to translate entirely” ^[2]. A replay built from sparse facial points may therefore look like a clean sentence that has lost its question or its negation, which is the most dangerous kind of clean.

So the stronger design still does not prove that meaning survived. It establishes what one layer of the system retained. That is a real advance over a polished back-translation, because it separates perception from translation and puts the machine’s evidence in front of the person whose language it is. It is not the same as verifying the signing, and nobody should be allowed to describe it that way.

A replay is still a visual channel

There is a second limit, and it is personal. I am DeafBlind, and I cannot see above eye level. A replay avatar placed in the upper part of a phone screen, rendered at low contrast or played at native speed, is a replay I may not be able to use. Other DeafBlind signers have different fields, different contrast needs, different preferred modes, and some use tactile signing, for whom a visual replay is not a verification route at all.

This is not an argument against replay. It is an argument against making replay the new universal precondition. A safeguard that moves the burden from reading English to watching an avatar has improved access for many signers and left others exactly where they were. The verification route has to be adjustable, with speed, scale, contrast, framing, repetition and close-up control, and there has to be another route when the replay is not accessible: a human interpreter, a tactile interpreter, or a Deaf reviewer the signer trusts.

The law already points the same way. Under the ADA’s Title II effective communication rule, a public entity must give “primary consideration to the requests of individuals with disabilities” when choosing an auxiliary aid or service ^[15], and a qualified interpreter is one “able to interpret effectively, accurately, and impartially, both receptively and expressively” ^[16]. A replay feature does not satisfy either provision and should not be marketed as though it did. DeepMind and AISLAC say the same of SL2T itself.

Deaf organizations have also been clear about avatars for a long time. The 2018 joint statement of the World Federation of the Deaf and the World Association of Sign Language Interpreters cautions against signing avatars as a replacement for human signers, particularly where information is live, complex or of significant importance, and accepts them only for pre-recorded static information where “deaf people have been involved in advising on the appropriateness of the signed sentences” ^[14]. A verification replay is a different use from the one the statement had in view: it shows a signer their own captured signing rather than delivering someone else’s message. But the principle carries over directly. Whether a replay is legible and trustworthy is a judgement fluent Deaf signers have to make, and the research on signing animation supports that caution, since facial expression measurably changes how well ASL animation is understood ^[13].

The evaluation gaps in the impact report matter here too. SL2T was “neither trained nor formally evaluated on signers under 18,” formal evaluation across signers with motor disabilities “remain[s] limited,” and benchmark disaggregation across sociolects such as Black ASL “remains an ongoing research priority” ^[2]. Whether a capture replay helps those signers detect errors is unknown, because whether the capture works well for them in the first place has not been shown.

Three objects, three claims

The design question becomes clearer once we stop treating every output as the same kind of record. A sign-to-text system that takes verification seriously handles three different objects, and each supports a different claim.

The signed source is the original communicative event, ideally kept as video when keeping it is lawful, proportionate, consented to and right for the context. It carries the richest evidence of what the signer produced, although interpreting it may still require a qualified human. The capture trace is the machine-readable representation retained from that event: landmark, motion and timing data. An avatar driven directly from the trace lets a signer or an evaluator inspect what the system kept and what it discarded. It is an audit artifact, not the original. The generated text is the model’s conclusion about meaning. It may be accurate, useful and editable. Until it is verified, it is a draft, and fluency does not promote it into a record.

Those distinctions change evaluation as much as they change interface design. A serious assessment compares all three objects rather than scoring only the last one, and Deaf-led testing can then ask separate questions. Did the sensing layer preserve the linguistically relevant signal? Did translation preserve the proposition and the intent? Could the signer detect a change and contest it? Was the verification route one this signer could actually use? Did the verification status stay attached to the text after it left the device?

The hierarchy among the three is the safeguard. The machine’s English is a proposal. The replay is an inspection path. The signed source, or a qualified interpreter’s rendering of it, is the authority. In “The Record Belongs to the Signer” I argued that the signed source should be the record of what a Deaf person said ^[5]. SL2T’s privacy design shows why that argument needs one more step. Where the source is lawfully discarded, the trace is the strongest evidence left, and a system that keeps neither the source nor a trace it can connect to the text has no basis for treating its English as a record of anything.

What the evidence would need to show

The capture replay is a strong design proposition, and so far that is all it is. Preservation may matter more than plausibility. That claim now needs evidence, and the evidence has a definite shape.

1. Measure correspondence

   Compare replays against source video for the same signing, rated by fluent Deaf signers who did not produce it, across lighting, framing, signing speed and occlusion. The question is how closely the replay matches what was signed, feature by feature.



2. Map what survives

   Test which grammatical features survive the 130-point, two-dimensional representation: handshape, location, movement, orientation, brows, mouth, head movement, contact and use of space. Publish the features that do not survive alongside those that do.



3. Test error localization

   Seed known errors and measure whether signers using the replay can tell a capture failure from a translation failure, and how often they are right.



4. Test correction across literacy

   Compare how many errors signers catch and correct with the replay, with English review alone, and with both, across a range of English literacy, so that the result shows whether the replay actually lowers the literacy precondition.



5. Test access configurations

   Include DeafBlind signers and test speed, scale, contrast, framing and repetition controls, with a tactile or human route recorded as the outcome where the replay does not work.



6. Test beyond the demo

   Run the tests with real signers across regions, registers and sociolects, including Black ASL, and in conversations rather than isolated sentences, and report results disaggregated rather than averaged.

Those questions do not diminish the idea. They are how an idea becomes assurance, and Deaf-led acceptance testing of exactly this kind is what the SLAT Index Evaluation Standard asks for before any deployment where an error can follow a person into a record ^[17].

What buyers should require

SL2T 1.0 is out of scope for the settings where these requirements matter most, and its developer says so. But sign-to-text will not stay in coffee shops, and other vendors will not all scope themselves as carefully. These are the terms a public body, an employer or a health system should put in a contract for any system that turns signing into text.

These are not accessories around the model. They decide whether an architectural limitation stays a limitation or becomes an institutional harm, and they are the part of the system a buyer actually controls.

Answering the question

Would the option to replay the system’s captured signing through an avatar solve the problem of verification in written English? It could materially reduce it, under four conditions: the avatar is driven from the pre-translation capture trace rather than generated from the output; the trace is rich enough to hold the linguistically relevant information for the use in question; the signer can compare the replay with what they meant; and another verification route remains available when the replay is incomplete or inaccessible.

It would not, by itself, establish that meaning survived. It would establish what one layer of the system retained. In SL2T as documented, that layer is 130 two-dimensional points with no tongue, no environment and no depth ordering, and the replay would faithfully show every one of those gaps as if they were not there. That is still a major advance, because it gives the signer the machine’s evidence in sign instead of the machine’s conclusion in English, and because it tells an evaluator where meaning changed. It is an inspection path, not a verdict.

The final safeguard is therefore not the avatar. It is the rule governing the relationship among the three objects. The machine’s English is a proposal. The replay is an inspection path. The signed source remains the authority, and when an institution cannot preserve that hierarchy, it cannot honestly claim to know what was said.

How this piece was checked