English is subject–verb–object. Korean is subject–object–verb. A correct translation moves words across cue boundaries — so translating each caption on its own produces subtitles that are both wrong and mistimed. This is the engine I built to fix that, for eight languages.
The verb decided is spoken first but must land last in Korean; the new budget is spoken second but must land first. Both cross a caption boundary, and the cue times change to match. Nothing here is illustrative — these are the timestamps the system emitted.
The intuitive way to translate subtitles is to walk the file and translate each caption. That works for language pairs that share a word order, and it breaks immediately for pairs that don't. Translate “The committee decided” on its own and you have to guess at a verb ending you cannot know yet, because the object it governs is in the next caption and hasn't been read.
Ask a language model to translate the whole file at once and the reverse problem appears: the prose is fine, but it is no longer attached to time. You have a paragraph where you needed a sequence of captions, each of which has to appear while its words are being spoken, stay up long enough to read, never overlap its neighbour, and fit on two short lines.
So the work divides cleanly in two. One half is ambiguous and needs judgement. The other half is arithmetic under hard constraints, where being approximately right is the same as being wrong.
Produce natural target-language text, and say which source cue each output word came from. Judgement calls with no single right answer — exactly what a language model is good at.
Cut that text into captions and assign start and end times that are monotonic, non-overlapping, inside the audio window, and readable. Guarantees, not preferences.
The bridge between the halves is a single idea: every output word carries an anchor — the moment its source audio was actually spoken. Because translation reorders, those anchors do not increase as you read. That is the whole problem, made into a number the algorithm can optimise against.
"""``anchor`` is the time (seconds) the source audio for this token was spoken. These need not increase with reading order: the verb's natural Korean position is at the end but its source audio was early.""" @dataclass class Token: text: str anchor: float break_cost_after: float = 0.0
Audio is stripped from video, downmixed to 16 kHz mono, split under the API's size cap, and transcribed. Output is an ordinary subtitle file, so everything downstream is unchanged.
WhisperA millisecond-accurate parser reads the file, then groups captions into whole sentences — the unit a translation can actually be correct about.
DeterministicOne call per sentence returns the translation plus a word-to-cue alignment, which becomes each token's anchor. Glossary terms are injected and checked.
ModelA dynamic program partitions the tokens into captions and solves for boundary times under hard constraints. No model involvement, and no randomness.
DeterministicBecause Stage 0 emits the same format the pipeline already accepted, a raw video file and an existing subtitle file are indistinguishable from Stage 1 onward. Speech recognition cost one new module and zero changes to the rest of the system.
The current Liontalk interface accepts either a subtitle file or a raw video. A video goes through Whisper, stops for transcript review, then enters the same timed editor as an existing subtitle file. These screens follow that current interface and use the real Persona and Attack on Titan samples from this case study.
The Persona sample starts with no subtitle track. The interface detects that it is a video, selects audio language detection and sends it to Whisper before translation.
All eight English cues are editable before translation. This is the checkpoint for correcting names, product terms and recognition mistakes before they propagate into every target line.
The video, source line, target line and timing checks stay together. A reviewer can change a cue, check its meaning, translate one unit again and download both the source and target VTT files.
These are short excerpts from the files I used while building the system. One begins with no captions at all. The other begins with an existing English subtitle file. Use the controls below each video to compare the source with the subtitle tracks produced during these runs.
A product walkthrough goes through Whisper first. I review the English transcript, then send the approved cues through translation, segmentation and timing.
This excerpt starts from the English VTT extracted from the source video. It skips speech recognition and enters the same translation and timing stages directly. The selector includes every supported target language from this run.
The checks cover monotonic timing, overlap, reading speed, line length, minimum duration and maximum duration. Every language in the selector passed those checks. Korean is the production language; Japanese, Spanish, French, Chinese, Russian and Arabic remain labeled Beta until their linguistic review sets are complete.
Where to cut a sentence into captions has no off-the-shelf answer, so I modelled it the way typesetters model line breaking. A Knuth–Plass-family dynamic program searches every partition of the token sequence and returns the one with the lowest total cost. Because a span's cost depends only on that span, the problem has optimal substructure and the DP solves it exactly rather than greedily.
def segment_cost(tokens, i, j, start, end, cfg) -> float: """Cost of making tokens[i:j] a single cue occupying [start, end]. Four competing terms. Local by construction, which is what gives the DP optimal substructure.""" cost = cfg.cue_penalty cost += cfg.w_reading * _reading_penalty(chars, duration, cfg) # too fast to read cost += cfg.w_line * _line_penalty(text, cfg) # too wide for the frame cost += cfg.w_split * _split_penalty(span, cfg) # cuts mid-phrase cost += cfg.w_drift * _drift_penalty(span, center) # drifts off the audio return cost
Reading speed, line length, split quality and timing drift pull against each other; the weights decide who wins. A second pass then solves for the boundary times themselves, where reading order is a hard constraint and timing fidelity is the soft one.
The payoff for keeping this half deterministic is that timing stops being a matter of opinion. Monotonic ordering, non-overlap and minimum on-screen duration are invariants the algorithm cannot violate, which makes them testable — and they are tested, on every run, against a fixed corpus.
The system started as English → Korean. Extending it could easily have meant forking the pipeline per language; instead everything language-specific lives in a profile: prompt wording, register options, reading-speed and line-length norms, the word-segmenter for scripts that don't use spaces, and whether the script runs right to left.
The deterministic machinery never learned a language. It scales per language, not per pair — only the prompt is pair-aware, and it is assembled from the two profiles. Adding a language is one profile and a quality-gate run, not a rewrite.
“The translation looks good” is not a claim anyone should accept, including me. So quality is scored by a gate that runs against a golden set of clips. One layer is structural and deterministic — timing invariants, reading-speed compliance, line-length compliance, glossary adherence. The second layer asks a model to score adequacy and fluency, which catches the failures arithmetic can't see.
A language is promoted out of Beta only when it clears every threshold. One of them doesn't, and it ships labelled Beta because of it.
| Target | Reading speed | Timing | Verdict | Ships as |
|---|---|---|---|---|
| Korean | pass | 1.00 | PASS | GA |
| Japanese | pass | 1.00 | PASS | Beta |
| Chinese | pass | 1.00 | PASS | Beta |
| Spanish | pass | 1.00 | PASS | Beta |
| Russian | pass | 1.00 | PASS | Beta |
| Arabic | pass | 1.00 | PASS | Beta |
| French | 0.875 | 1.00 | FAIL | Beta |
French fails on reading speed at 0.875 against a 0.90 threshold. The cause is intrinsic: French renders the same meaning in more characters, so more captions run past a comfortable reading rate in the time the audio allows. It is documented rather than papered over, and it stays in Beta until condensation improves.
Underneath the gate sits an ordinary test suite that runs entirely offline — a static translator stands in for the model, so the whole pipeline, service and job lifecycle are exercised with no API key and no network.
The engine is a pure library with no web framework in it. A FastAPI service wraps it and a Next.js editor consumes a frozen, typed contract, so all three evolve independently.
Re-timing edited text re-runs only the algorithm, never the model — so corrections are instant and free. A single caption can be re-translated on its own for one call.
Speech recognition stops at a review step. Nothing is translated until a person has approved the transcript, because translation quality is capped by transcript quality.
A glossary is injected into the prompt and checked afterwards, flagging terms that didn't survive. A whole brief can be saved and reused across episodes.
Live state serialises losslessly to SQLite or Postgres, and a startup sweep fails anything orphaned by a dead process instead of leaving clients polling forever.
Routing by model name drives any OpenAI-compatible endpoint, so the same code runs against a frontier model or a budget one, chosen per job.
Both file and media uploads are priced up front from token and duration estimates, with no inference call needed to produce the quote.
The interesting decision in this project was not which model to call. It was working out which half of the problem a model should be nowhere near — and then building that half properly, as an algorithm with guarantees I could test, measure and report honestly, including where it falls short.