# Jot for Windows → Mac & iOS: engineering handoff

**Audience:** the agents building Jot for Mac and iOS.
**Purpose:** re-implement three features the Windows app now gets right:
1. **custom vocabulary**, where the Mac/iOS apps still have the older bugs;
2. **real transcription progress**;
3. **speaker labels** (Nemotron 3 Diarization).

- Reference implementation: `github.com/vineetu/jot-windows` (private), at commit `be0d455` or later. Windows file and function names are given so you can ask for the code.
- Everything runs on device. Numbers are measured unless marked *unverified*.
- Recommended order: vocabulary first (it fixes user-visible bugs today), then progress, then speaker labels. Speaker labels depend on the word timings described in §3.5, and on the long-audio fix in §3.6, which Mac/iOS very likely need anyway.

---

## 1. Custom vocabulary

Constants are quoted from the Windows code. Anything marked **unverified** was not confirmed in code or measurement.

### 1.1 What the user sees

- **Vocabulary page.** It has its own entry in the left pane (`Views/VocabularyPage.xaml`, `ViewModels/VocabularyViewModel.cs`).
  - Subtitle: *"Terms Jot should prefer when it transcribes. Kept on this PC. Works best under about 100 terms."*
    - Non-English covered languages append *" · spelling matching only"*.
    - Languages that are off append *" · not active in this language."* plus one of three reasons:
      - Auto detect: "can't tell which language… pick one in Settings → Language".
      - A list exists but measured too weak (Slovenian): "can't yet tell your terms apart from ordinary X words reliably enough to be safe".
      - No list at all: "has no everyday-word list for X".
  - A term is **just the word** ("Term / How you want it spelled"). There is **no "sounds like" field** (08f8268).
  - Learned spellings show on the row as *"Jot learned: a · b"* and can be removed in the edit form. *"Renaming clears what Jot learned about this term."*
  - Row warnings:
    - ≤ 2 chars: "Too short".
    - More than 4 words.
    - Every word is common: "Common word — Jot won't swap this on its own. You can confirm it on the recording."
  - Cap `MaxTerms = 200`. At the cap the Add button is disabled; terms are never silently dropped.
  - Pasting a list offers "Add N terms" or "Add as one term". Delete has Undo, and a search box appears on long lists.
- **No on/off switch** (80017a2). With no terms nothing runs.
- **Pill mark.** It shows after delivery for **applied** corrections only, de-duplicated by (original, term).
  - One correction shows the term, truncated to 13 chars + "…" past 14. Two or more show the count.
  - The expand list shows up to 5 lines of `orig → term`, then `+N more`.
- **Fix a word.** Right-click a selection in a transcript and choose **"Add to Vocabulary…"**. The dialog has "When Jot hears" (pre-filled) and "Spell it as" (focused). On Add:
  1. Replace the **whitespace-trimmed** selection.
  2. If every word of the term is common, stop at "Fixed here. "X" is an everyday phrase, so it wasn't added."
  3. Otherwise `AddMapping(heard, term)`, which creates the term or appends a learned alias.
  4. Then `CorrectionStore.Adjust(heard, term, +1)`.
- **Ask card and question sound.** These appear only when the deck is non-empty.
  - The deck holds at most `MaxAsks = 3` **applied** corrections, excluding pairs already answered.
  - A "hm?" motif plays (G4→E5; setting `SoundQuestion`, default on). Then an **activatable** window opens. The pill stays non-activating.
  - Card text: "Jot wrote {Term}.", an "N of M" counter, and two chips: original, and `term ✓` (focused).
  - Keys: ←/→ choose, Enter confirm, Esc skip. Each card counts down 8,000 ms.
  - If card 1 times out with no interaction, the **whole deck** is skipped. After any interaction, a timeout skips only that card. Ignoring a card is never recorded as an answer.
  - Every exit delivers. The paste is held until the deck resolves, then forced to the **origin window**. The done chime is suppressed after a question.
- **Review block in recording detail.** Header "Jot guessed on N words".
  - One row per **occurrence**, with a CHANGED or KEPT badge, context, and original/term chips.
  - A pick edits the transcript at that occurrence, flashes it, and moves learning.
  - Rows whose anchor can't be resolved strictly are hidden, never nearest-matched.

### 1.2 English pipeline end to end

Order at stop (`RecorderController.StopAndDeliverAsync`):

1. Parakeet TDT decode **with phrase bias**.
2. `TextPipeline.Clean`.
3. **Vocabulary block**: corrector → gate → provenance → ask. It must be the **last** transform.
4. Save, `Commit(id)`, bank the ask answers, then paste.

#### 1.2a Decode-time phrase bias (`Transcription/PhraseBias.cs`, `ParakeetTranscriber.DecodeGreedy`)

**Input.** The saved term **text only**; learned aliases are not biased. It is re-read per utterance and rebuilt only when the list changes. It never throws: if the provider fails, that utterance gets no bias.

**How a term becomes something to match.** Normalise each term to `" " + whitespace-collapsed lowercase`; the leading space marks a word start.
- Drop terms under 3 chars (trimmed). Remove duplicates.
- For every term and offset `o`, precompute `edges[term][o]`: every vocab piece whose lowercased text (▁ → space; excluding `<…>` controls and blank) equals `term[o : o+len(piece)]`.
- So matching is on **characters, not one tokenisation**, and it is case-insensitive. The gate restores the saved casing later.

**Score, per step.**
- Compute a log-softmax over the token logits.
- Candidates:
  - continuation edges of active matches, bonus `ContinueBonus = 3.0`;
  - when starts are allowed, every offset-0 edge, bonus `StartBonus = 2.0`.
- A candidate needs `logp ≥ −10`, and it wins only if `logp + bonus > logp(argmax)`.

**Start rule.**
- `allowStart = argmax not blank AND argmax piece starts with a space AND not in cooldown AND (t, tokenCount) not banned`.
- So a term can never come out of silence and never split a word.

**Advance.**
- After each emitted non-blank token, all active matches advance on it, and new matches open at offset 0 (also when the model emits an offset-0 piece unbiased).
- Reaching the term's end sets `completed`.

**Speculation and rollback.**
- At the first chosen token ≠ argmax (with no speculation open), snapshot: `t`, token count, both LSTM states, the emitted-this-frame count, the active set, and the key `(t, tokens.Count)`.
- If `completed`, drop the snapshot and keep the term.
- If a non-blank token is emitted and the active set becomes empty, **rewind unconditionally**:
  - restore state;
  - **clear the active set** (abandon the match, don't restore it);
  - ban the key, cool down until an unbiased token, and count the rewind.
- If the utterance ends inside a speculation, rewind.
- `MaxRollbacks = 64` per utterance. When it is reached, **biasing is switched off**; rollback never is.
- Net property: each biased stretch outputs the **whole term** or **exactly what the model would have written unbiased**.

**Known limits.**
- Knock-on edits: the prediction state differs after a biased token, so the next word or two can change (≈ 40 of 140 target changes on real dictations, mostly punctuation and fillers).
- A suffix after the term isn't checked (**unverified**).

#### 1.2b Post-pass corrector (`Vocabulary/VocabularyCorrector.cs`)

This also runs in English, as a floor under the bias.

**Forms.** Each term contributes its text, its aliases, and (for a multi-word term) its space-stripped merge ("Ramaa Nathan" → "RamaaNathan"). Forms under 2 chars are skipped.

**Skeleton.** NFC, then lowercase, then keep **alphanumeric Unicode scalars only**: letters, **marks**, and numbers.

**Windows.** Word n-grams of width 1 to `min(form word count + 1, 8)`. An acronym form may use width = its letter count ("W A S A P I").

**Distance and acceptance.**
- Restricted Damerau–Levenshtein (OSA) over scalars.
- Edit budget: 0 if the skeleton is ≤ 3, else `ceil(len/5)`.
- `edits / max(len)` must be ≤ the per-language maximum (§1.3).
- A cheap pre-filter rejects on length difference and letter bitmasks first.

**Choosing matches.**
- Keep the best match per window. A tie between **different terms** on one window drops it as ambiguous.
- Placement:
  1. **Claim identity matches first.** A casing-only single word is still emitted so the gate can re-case it.
  2. Take the rest by distance ascending, then width descending, then index ascending, without overlap.
  3. A window wider than the form's word count is allowed only if `Gap ≤ 0.45` ("width unlock").
- Detections are encoded as a time `(i + w/2)/nWords × duration` for the gate's positional search.

#### 1.2c Safety gate (`VocabularyGate.ApplyFromDetections` + `Decide`)

**Placement.** Candidate hosts are contiguous windows up to the maximum word count of the term or its aliases. A one-word term may widen up to its letter count, but only if the concatenated skeleton **equals the term exactly**.
- Valid window:
  - no claimed words;
  - plain spaces only (a newline is a boundary);
  - no inner punctuation;
  - it aligns uniquely, or it is an exact concat split.
- A candidate needs `PlausibilityGap ≤ 0.45`, where the gap is the smallest normalized Levenshtein distance against the term or any alias.
- Pick the nearest position; on a tie, the wider window wins.
- **Promote** a partial pick to the best containing window.
- Claim the chosen words.

**Per pick, in order.**
1. Overlap with already-published text: skip.
2. **Dedup absorb.** For a multi-word term whose trailing words repeat after the host, widen to absorb them. If the host ends on punctuation, force BLOCK.
3. **Partial-term skip.** If the span has fewer words than the term and is closer to one of the term's own words than to the whole term, skip.
4. **Apostrophe guard.** If the span has `'`, `’` or `ʼ` and neither the term nor any alias does, BLOCK.
5. **Inflection guard.** Single words only. BLOCK when all of these hold:
   - they share a stem of at least 4;
   - each tail is at most 3;
   - the skeletons differ;
   - the span skeleton (length ≥ 7) equals a common word minus its last character.
6. **Identity or casing.**
   - Identical: no-op.
   - Same letters but different casing: publish the saved casing, keep edge punctuation, no proposal.
7. A multi-word term must **align uniquely** to its span, or BLOCK.
8. **`Decide`**, with confidence "unknown" (0.85):
   - **(0) Override.** net ≤ −1: BLOCK. `AlwaysReplace`: OVERRIDE. net ≥ 1 and the original is not common: OVERRIDE.
   - **(1)** Gap > 0.45: BLOCK.
   - **(2)** Multi-word term: APPLY.
   - **(3)** Confidence ≥ 0.95 and margin ≤ 4.0: BLOCK (never fires on this path).
   - **(4)** **Any** word of the span is in the ~24k common list: BLOCK and mark as an ask candidate. Exception: a multi-word span whose concatenated skeleton is exactly a one-word term ("Mid East" → "Mideast").
   - **(5)** Otherwise APPLY and mark as an ask candidate.
9. On APPLY, splice the term, keeping edge punctuation. Record a `Proposal` with **grapheme-cluster** offsets.

#### 1.2d Learn loop, storage and provenance

**Spellings are learned, not typed.** Aliases come only from corrections, and an alias belongs to exactly one term.

**`SanitizeTerm` is the single choke point** for every writer. It applies NFC, strips `: , ;` and control characters, collapses whitespace, strips a leading `#`, and **never case-folds**.

**Files** (`<data>/Vocabulary/`; include the folder in erase-all-data and data-folder moves):

- `vocabulary.json`:
  ```json
  {"version":1,"terms":[{"id":"<guid>","text":"Sriram","aliases":["shiram"]}]}
  ```
- `corrections.json`, keyed by `lowercased(term)`:
  ```json
  {"sriram":{"mappings":[{"originalWord":"shiram","term":"Sriram","confirmations":1,"reverts":0,"blockedKeeps":0,"alwaysReplace":false}],"suppressedBlocks":["shiram"],"mergeAskedPhrases":[]}}
  ```
  - `originalWord` is normalized: NFC, lowercase with İ→i̇, whitespace collapsed, outer ` .,!?;:"'()` trimmed.
  - `net = confirmations − reverts`. A negative adjustment clears `alwaysReplace`.
  - If the file exists but can't be decoded, **disable persistence; never overwrite it**.
- `provenance/<recordingId>.json`: `{records[], verdicts{recordKey→"term"|"original"|"alt0"}, contributions{mappingKey→±1}, anchoredText}`.
  - `recordKey = lower(orig)|lower(term)|originalStart`.
  - Use a pending slot before the id exists; `ClearPending()` at dictation start and `Commit(id)` after save.
  - Reconcile anchors by diffing `anchoredText` against the live transcript.

**Learning rules.**
- At most ±1 per mapping per transcript: a "term" pick is +1, "original" on an applied row is −1, mixed is 0.
- Ask answers go through `SetVerdict` → `Adjust`, then **`SuppressBlock(pair)`**, so the pair is never asked again.
- The ask deck is drawn from **applied records only**, ranked by net.

### 1.3 Non-English

- **Model-free corrector plus the gate only.** No download.
- `Textual` mode only when a common-word list exists **and** the language is in the shipped table. Everything else is Off.
- **18 languages** and their maximum distance:
  - uncapped (1.0; the budget tops out around 0.33): cs, da, es, fi, fr, hu, it, nl, pt, ro, sk, sv;
  - tightened: bg 0.25, de 0.15, el 0.20, pl 0.20, ru 0.20, uk 0.20.
  - English is uncapped.
- **Excluded:**
  - sl: 1.72 false applies per 1k words adversarial; the only fix recovers 16%; 58.6% WER.
  - sr: no locale can select it.
- **Why (E6, 8f67647).** Measured on ~9.5–10k FLEURS clips, with a budget of ≤ 1.0 false applies per 1k words on realistic (25-term) and adversarial lists.
  - The brake's strength is the list's coverage of word types: en 88.8%, el 56.9%.
  - All 531 blocks were the common-word brake, and the 0.45 ceiling never fired. So **the per-language knob is the corrector distance**, not the gate.
- **Fail closed** (22865d4). Unknown subtags get `Unmeasured = 0.15` (the tightest), never English's uncapped setting.
  - **Auto detect is Off** (the engine discards its detected language).
  - Residual quirk: the mode normalizer maps junk to en-US (the app only passes canonical codes).

### 1.4 Bugs found and fixed (each one is a checkable symptom)

1. **"name" → "Jamy", "and" → "Andre"** (the original Mac bug). Cause: detector output applied with no gate. Fix: the gate plus the common-word brake (070ef9b). *Check: add "Jamy" and dictate "name"; it must stay.*
2. **Spanish "lista" → "Lisa".** Cause: running without a language list. Fix: no list or no measurement means Off (070ef9b, 51270f0). *Check: Auto detect and languages without a list never change text.*
3. **"Claude code" → "Claude Code code".** Fix: dedup absorb, and block if the host ends on punctuation (070ef9b).
4. **"I love cloud code." never corrected.** Cause: single words compared against the whole multi-word term (0.60 > 0.45). Fix: N-word windows (0.20) (070ef9b).
5. **"Claude wrote…" → "Claude Code wrote…".** Cause: inserted words that were never said. Fix: partial-term skip (9398e62), promotion (8194d62, 28c5f45), overlap drop (8ef5fa5).
6. **"Ramanathan" replaced by "Ramaa".** Fix: feed the space-stripped merge as an alias.
7. **Possessives rewritten ("Mariana's" → "Marianas").** Cause: the skeleton drops the apostrophe. Fix: the one-way apostrophe block (51270f0). This was 5 of 20 false applies.
8. **"George W" → "George" (the initial deleted).** Fix: claim identity matches first (51270f0). FP-overwrote went from 10 to 0.
9. **Inflections flattened** (ru/el/pl/uk). Fix: the stem-minus-one guard (bd211a0). Greek went from 1.42 to 0.89 per 1k.
10. **Split one-word terms never corrected** ("Mid East", "nemo tron"). Fix: exact concat only (d075005, b9f8b1f, 74e456f, 3e58ed3). 2,003 near-concats stay blocked.
11. **Casing never fixed ("sriram").** Fix: publish the saved casing (a0fdbec).
12. **Silent zero recall from casing** (904b4df). A term typed ALL CAPS got 0/65, and Title Case lost 32 points. **Port lesson:** lowercase both terms and pieces, and restore casing in the gate. *Check: "KUBERNETES" and "okta" must both be found.*
13. **Unknown language got English's loosest setting.** Fix: fail closed at 0.15 (22865d4).
14. **Corrector shipped to 19 languages on English numbers.** Fix: per-language distances; Slovenian cut (8f67647).
15. **Naive shallow fusion wrote fragments** ("and AMD" → "NEMD", "Okay" → "Ok"). Fix: character matching, word-start-only starts, rollback (fca9e13).
16. **Half-matched "chat GPT" leaked.** Cause: rollback stopped when its budget ran out while biasing continued. Fix: the budget disables *biasing*; undo is unconditional (d2d04df).
17. **Rewind storm (64 on 11 s).** Cause: a rewind restored the failed match. Fix: abandon it, ban `(t, tokenCount)`, cool down. Worst clip: 20 rewinds (d2d04df).
18. **Terms started mid-word.** Fix: start only when the argmax is a non-blank word-start piece (d2d04df).
19. **A hang stalled dictation until restart.** Fix:
    - `VocabularyDeadlineMs = 4000` via WhenAny, with ungated text delivered on any failure;
    - the ask deck backstopped at `cards × 8000 + 2000` ms;
    - a `finally` that closes the card (070ef9b).
20. **Paste went into the ask card.** Fix: force the origin window once the ask block starts, including on error.
21. **Held paste never landed** (Mac). Fix: every exit completes the one resolution.
22. **Asked forever.** Fix: an applied-only deck, and answered pairs added to `suppressedBlocks`.
23. **A switch that only meant "ignore my terms".** Fix: always on (cost −2% to +0.7%, within noise) (80017a2).
24. **"Sounds like" field plus an acronym nag.** Fix: learn aliases from corrections only (08f8268).
25. **Typed `:` `,` or a leading `#` corrupted the file, and an editor bypassed the sanitizer** (iOS). Fix: one `SanitizeTerm` at the store.
26. **"My name is VineetSriram."** Cause: a double-click selection included a trailing space. Fix: trim before splicing (commit **unverified**).
27. **Unicode fidelity.** Keep combining marks in the skeleton; handle Turkish İ; don't fold Greek final sigma (070ef9b).
28. **Common-words list missing from the bundle** (Mac's "#1 shipped bug"). This silently disables the brake. Embed the lists and test that each one loads.
29. **Review rows edited the wrong occurrence after Replace All** (iOS). Fix: fingerprint and diff reconcile; hide unresolvable rows.

### 1.5 Validation

- **Golden fixtures** from jot-shared @5326460, replayed: gate decide, detections apply, store round-trip, ask policy, provenance verdicts, key normalize, pair key.
- **E5** (FLEURS en_us: 1,041 clips, 22,490 words; 214 missed-term chances). Corrector recall was 36.9% at 0.93 false applies per 1k (155 terms), and 34.1% at 0.27 per 1k (25 terms).
- **E6** per-language (§1.3). **E7** inflection guard: false applies 64 → 58 and 146 → 135.
- **Concat scan:** 2 exact windows, both correct, 0 false.
- **Phrase bias:**
  - TTS: "Vineet Shiram" → "Vineet Sriram"; the "beneath" decoys stayed byte-identical at every strength.
  - Real Mac export (55.7 h, 25 terms, time-split holdout):
    - decoys: 297 clips, 9,963 words, 1 change, which was correct;
    - targets: +96 correct mentions and ≈ 3 wrong insertions (≈ 0.1 per 1k).
  - "Okta" went 39 → 57.
- **Not measured:** bias plus corrector together on FLEURS; multi-speaker tuning.

### 1.6 Port checklist

- [ ] Phrase bias:
  - character-level edges over lowercased pieces; terms ≥ 3 chars, text only;
  - +2.0 start, +3.0 continue, −10 floor;
  - start only at an argmax word-start piece.
- [ ] Rollback:
  - snapshot at the first override;
  - unconditional rewind; **abandon** the match; ban `(t, tokenCount)`; cooldown;
  - rewind at end of utterance;
  - 64 rewinds → biasing off, never undo off;
  - monotonic progress.
- [ ] Corrector:
  - OSA distance, `ceil(len/5)` budget, exact match at ≤ 3 chars;
  - windows up to word count + 1 (acronyms: letter count), max 8;
  - claim identity matches first; drop ambiguous ties;
  - per-language maximum distance.
- [ ] Gate:
  - window hosts, promotion, dedup absorb and punctuation block, partial-term skip;
  - apostrophe guard, inflection guard;
  - casing publish, alignment;
  - `Decide` 0–5, 0.45 ceiling, any-word common brake, exact-concat exception.
- [ ] Skeleton = NFC-lowercased alphanumeric scalars, including marks. Keys handle İ.
- [ ] Languages: en plus the 18; sl and sr off; Auto detect off; unknown values fail closed.
- [ ] Vocabulary runs last, under a 4 s deadline and try/catch, delivering ungated text on failure.
- [ ] Ask card:
  - applied-only deck, ≤ 3 cards, 8 s each;
  - card-1 timeout skips the deck; every exit delivers;
  - paste to the origin window; question sound;
  - answered pairs suppressed.
- [ ] Learning: aliases only from corrections; +1 on Add; ±1 per mapping per transcript; one revert clears always-replace; never overwrite an unreadable ledger.
- [ ] Every writer goes through `SanitizeTerm`. No on/off switch, no "sounds like" field.
- [ ] Provenance uses **grapheme-cluster** offsets, fingerprint reconcile, and strict resolution.
- [ ] Turn §1.4 into tests, especially items 1, 3, 5, 7, 8, 12 and 16–18.

---

## 2. Transcription progress (a real bar, not a spinner)

### What the user sees
- On stop, the pill shows **"Transcribing… 42%"** over a thin accent bar that tracks the real work. It never goes backwards and never shows 100% early.
- Speaker detection uses the same look in-page: "Finding words… 25%", "Downloading the speaker model…" with download progress, "Finding speakers… 80%".
- **Success chime rule:** play the success sound only when stop→delivery took **≥ 5 s** (`SuccessChimeAfterMs = 5_000`), and not when a vocabulary question card was shown (the question has its own sound). Short dictations stay silent.

### How it works (Windows: `ParakeetTranscriber.TranscribeWindow`, `DecodeGreedy`, `Learn`, `MonotonicProgress`; `PillController.OnTranscribeProgress`)
Parakeet has three stages: preprocessor (fast), encoder (one opaque ONNX call, the long part), and the TDT greedy decode loop (a loop we control).

1. **Split the bar by learned share.** `share` = the fraction of total time the encoder usually takes (starts at **0.6**). The encoder fills `[0, share]` and the decode fills `[share, 1]`.
2. **Encoder: time-based estimate.** The encoder can't report progress, so a 100 ms timer reports `share × min(0.95, elapsed / expected)`, where `expected = audioSeconds × msPerAudioSecond`. The rate starts at **60 ms per audio-second** and is learned. The 0.95 cap means a slow run never shows its slice as done.
3. **Decode: real progress.** Report `share + (1 − share) × t / validFrames` from the frame index `t`, only when the integer percent changes.
4. **Learn after each run** (runs of 1 s or more only, since fixed overhead dominates below that). This is an EMA with **α = 0.3** of `encoderMs / audioSeconds` and of `encoderMs / (encoderMs + decodeMs)`. The first run seeds the values directly.
5. **Monotonic wrapper.** Clamp to [0, 1] and drop any value ≤ the last one reported. The encoder timer and the decode loop race, and without this the bar jumps back.
6. **Long audio (windows, see §3.6):** each window gets a slice of the outer bar proportional to its length, including its lead-in. One outer monotonic reporter covers the whole run.
7. **UI:** marshal to the main thread, and **drop reports once the recorder has left the Transcribing state**. Otherwise a late report paints a bar over the finished transcript.

Port notes: on Apple, CoreML/ORT encoder calls are just as opaque, so use the same time estimate. Keep the learned rate per device (it varies a lot between an M-series Mac and an older iPhone). Persisting it across launches is optional; Windows keeps it in memory only.

---

## 3. Speaker labels (diarization): NVIDIA Nemotron 3 Diarization

### What the user sees
- In a recording's "…" menu, **Detect speakers**. It's available when the recording has audio and word timings, or when the English engine can make them. Progress is shown as in §2.
- The transcript becomes **turns**: a coloured bar, the speaker's name, the start time, then the text. **Show plain** switches back.
- **Name boxes** (one per speaker, with a colour dot) rename every turn at once. The model only separates voices, so names default to "Speaker 1…N" in order of first appearance.
- The **playback bar becomes a speaker timeline**: coloured stretches per speaker, with gaps under 0.5 s bridged. Hovering shows "Name · 0:06–0:11". **Click or drag to seek.**
- Labels are **saved with the recording**. They survive transcript edits, and they survive audio deletion under retention.
- **WebVTT export** uses real timings with `<v Name>` voices.
- Palette (8, index = speaker): `#4C8BF5 #2EA66F #E8912D #9B6BDF #E0525A #1FA3B3 #D65DB1 #9AA33A`.

### 3.1 Model
- `nvidia/Nemotron-3-Diarization`: 100M parameters, Sortformer-style, up to **8 speakers**, arrival-ordered outputs, licence **OpenMDW-1.1**.
- ONNX export: `onnx-community/Nemotron-3-Diarization-ONNX`. **Their int8 and q4 files use 8-bit `MatMulNBits`, which older ONNX Runtime (1.20) cannot load.** We re-quantized their fp32 export with `onnxruntime.quantization.quantize_dynamic` (MatMulInteger + DynamicQuantizeLinear): **98 MB, 33× realtime on 2 CPU threads**, 99.84–99.98% identical frame decisions to the official fp32 model.
- Hosted, SHA-pinned, on `github.com/vineetu/jot-models`, release **`jot-model-nemotron3-diarization-int8-v1`**. Download on first use (most users never need it):

| file | bytes | sha256 |
|---|---|---|
| `model_int8.onnx` | 467,029 | `1851b7fa1e9e814b25fc89b6c3898761f75393e69b8a399613b6692ccb4e719c` |
| `model_int8.onnx.data` | 103,648,768 | `47c6da26b249151bbb9e7100dbdc568e550b2e2d80981a256b43a4a299c7e692` |
| `config.json` | 1,623 | `a758bf59867e27a7d92cf7a9149cae6dfc212b408b2ba74bfb87994708a803c5` |

- ONNX I/O: inputs `input_features [1, 8n, 128]`, `cached_embeds [1, k, 512]`, `attention_mask [1, k+n]` (int64). Outputs `logits [1, (k+n)·8, 8]` (10 ms frames), `chunk_embeds [1, n, 512]`, `silence_embeds [512]`.
- Apple option: the same ONNX runs in ONNX Runtime for iOS/macOS. A CoreML conversion is possible but unverified; if you convert, re-check fidelity against the fixture (§3.7).

### 3.2 Features: do not reuse Parakeet's preprocessor
Parakeet's `nemo128.onnx` **normalizes per feature**, and this model expects **un-normalized** log-mel. Compute (Windows `DiarizationFeatures.cs`):
- Pre-emphasis 0.97: `y[0] = x[0]`, `y[i] = x[i] − 0.97·x[i−1]`.
- Centred STFT: pad 256 zeros on each side. n_fft 512, hop 160, **symmetric** Hann of length 400 placed at offset 56 inside the 512 window. Take the power spectrum `|X|²`.
- 128 **Slaney** mel bands (librosa `mel(sr=16000, n_fft=512, n_mels=128, fmin=0, fmax=8000, norm="slaney")`). Then `log(mel + 2^-24)`.
- Frames = `1 + samples/160`. **Only `samples/160` are valid: zero the last frame** (the reference processor masks it).
- Our bug: a static constant used before it was initialized gave NaN features. Check that there are no NaNs.

### 3.3 Offline chunk loop (Windows `NemotronDiarizer.Diarize`)
Read `config.json`. The **top-level** `chunk_length` (340 encoder frames = 27.2 s), `chunk_right_context` (40), `fifo_length` (40) and `speaker_cache_update_period` (300) override the streaming ones for offline use. `speaker_cache_length` is 264.
1. Pad features to a multiple of 8 (subsampling), giving `nEmbeds` encoder frames.
2. For each chunk `[start, end)` of 340 frames, plus up to 40 look-ahead frames: `cached = cache.GetEmbeds()` (the `[speakerCache, fifo]` concatenation).
3. `attention_mask[i] = 1` for cached frames. For chunk frames it is `(start + i − numCached)·8 < validFrames`, which masks the padded tail.
4. Run the model. Keep the logits of the **chunk's own frames only**, `[(numCached)·8, (numCached+nChunk)·8)`.
5. `cache.Update(inputEmbeds = cached ++ chunk_embeds, logits, silence, nChunk, mask)`.
6. At the end, take sigmoid for per-speaker probabilities per 10 ms and **zero frames ≥ validFrames**.

### 3.4 Speaker cache (the host owns it; a line-by-line port of Transformers' `Nemotron3DiarizationSpeakerCache`)
Windows `SpeakerCache.cs`, batch 1. The essentials, where each one was a real bug risk:
- Pool probabilities: sigmoid, then average each group of 8 logit frames (floor). **Zero pooled probabilities where the step mask is 0.**
- Append the chunk's frames to the FIFO. If the FIFO exceeds 40, pop `min(max(300, len − 40), len)` frames into the speaker cache along with their probabilities. Use the **stored** probabilities once the cache has been compressed, and the fresh ones before that.
- Compress when the cache exceeds 264: per-frame score `log(max(p,τ)) − log(max(1−p,τ)) + Σ_s log(max(1−p_s,τ)) − log 0.5` with τ = `prediction_score_threshold` (0.25). Non-speech frames (`p ≤ 0.5`) score −∞. A speaker with ≥ `min_positive_scores` positive frames drops its weak ones. Add `latest_frames_score_boost` to just-popped frames. Boost each speaker's top-k: strong `−2·log 0.5`, weak `−log 0.5`. Add 1 silence slot per speaker at +∞. Take the top 264 over **speaker-major** flattened scores. −∞ picks map to the silence frame. Sort the indices to keep order.
- Verify against the reference: 99.999% identical decisions on a 17.5-min meeting (NumPy mirror) and ≥ 99.9% for the C# port against the official output.

### 3.5 Word timings: from Parakeet, recorded at transcription time
Speaker labels are per **word**, so the ASR must say when each word was spoken.
- In the TDT greedy loop, store **`(tokenId, frameIndex)` pairs in one list**, so a rollback of biased speculation truncates both together (with two lists they drift apart).
- One frame is 80 ms. A word starts at a token whose piece begins with a space (`▁`); **digits come as a bare space piece followed by the number**, so a lone space also starts a word. A "word" that is only punctuation is merged into the previous word. (Our first version produced "Version 1.4.3" as one word.)
- Word end = `min(next word start, last token frame + 80 ms, start + 1 s)`, at least 80 ms, so a word before a long pause doesn't claim the pause.
- The text must be **byte-identical** to the untimed path (we verified this on 61 clips).
- Save a sidecar `<audio>.words.json` = `{"V":1,"Words":[{"T":"Hello,","S":0.32,"E":0.8},…]}` at dictation **and import** time. Delete it together with the audio under retention. If there's no sidecar, re-transcribe the audio (English engine only) when the user asks for speakers.
- **Imports must keep their audio** (16 kHz mono WAV) or they can never get speakers. Windows imports previously discarded it.

### 3.6 Long audio: a Parakeet bug you probably have too
Parakeet TDT v2's encoder has a **5000-frame relative-position table**, so one pass fails beyond **~400 s (6.7 min)**. On ORT the error is a broadcast failure in `layers.0/self_attn/Add_2` ("1335 by 6335"). On Windows, **every English recording or import over ~6.7 min failed** until this fix. Fix:
- ≤ **300 s**: one pass, unchanged.
- Longer: `ceil(len / 240 s)` windows. Move each cut to the middle of the **quietest 300 ms within ±10 s** of its target (10 ms energy frames, prefix sums).
- Each window after the first also decodes **10 s of lead-in** before its cut and keeps only tokens from the first **word start at or after the cut**. Starting cold made the model drop a whole sentence ~20 s into the window; 2.7 s of lead-in was already enough.
- Short windows hurt: 90 s windows dropped sentences. Keep windows as long as is safe.
- Take the inference lock per window, so a dictation can run between the windows of a long import. Check cancellation between windows.
- Result: 17.5 min in ~70 s. Seams are clean (no lost or repeated words). The remaining rare dropped sentence occurs mid-window and appears in single-pass decoding too.

### 3.7 Speaker per word (Windows `SpeakerLabels.Assign`)
- For each word, take the mean probability per speaker over its frames `[start, end)`. Pick the highest if its mean is ≥ **0.2**. Otherwise retry with ±0.5 s padding. Otherwise mark it unknown.
- Unknown words take the previous word's speaker (leading unknowns take the first known).
- **Fold rare speakers:** a speaker with fewer than 3 words **and** under 1 s in total gets reassigned to neighbouring words, unless it's the only voice.
- **Renumber** by first appearance, so "Speaker 1" is whoever talks first.
- Measured: 100% of words correct on 2-speaker test conversations and a solo clip, **99.9% on AMI ES2004a** (17.5 min, 4 speakers; overlapping speech excluded). The diarizer takes ~29 s for 17.5 min on a laptop CPU. No smoothing was needed.

### 3.8 Storage and alignment to an edited transcript (Windows `SpeakerAligner`)
- Store on the recording: `SpeakerData { Words[], Speakers[], StartMs[], EndMs[], Names[] }` in parallel arrays. These are the **engine's** words, not the displayed text. Serialize compactly on one line, even inside an indented library file (~40 KB for 17 min). Treat damaged or mismatched lengths as "no speakers".
- At display time, **align the current transcript** (after cleanup, number formatting, vocabulary fixes and user edits) to the stored words. Use a Myers diff on case- and punctuation-blind keys, with common prefix and suffix trimmed first. A matched word takes the detected speaker and time. An unmatched word takes the previous word's speaker, with its time just after the previous word. Above **2048 edits** (a wholesale AI rewrite), map by position instead.
- Consecutive words with the same speaker form a turn. Renames edit only `Names[]`.
- Don't hold strong references from the long-lived recording to the page (Windows uses a weak event subscription). Our first version went silent when the page got an unload/load cycle.

### 3.9 Job, UI and VTT details
- One detection job per recording, owned by a **service**, not the page, so leaving the page neither cancels it nor loses the result, and coming back shows the same progress. Apply the result and failures **on the main thread**. We had a job that hung forever because a throw happened before the `try` of a fire-and-forget task.
- Progress: 0–50% "Finding words…" (only when re-transcribing), then "Downloading…" if needed, then "Finding speakers…".
- VTT: a cue per speaker change, split at a sentence end once a cue passes 7 s, hard stop at 12 s. Timestamps are `HH:MM:SS.mmm`. Escape `& < >` in text and in voice names. Without timings, one cue spans the recording.
- Seek bar: click/drag → fraction × span → player position. A seek before the audio is opened must be applied when it opens.

### 3.10 Validation you should reproduce
- **Fidelity:** features vs librosa (tolerance 2e-3 on the first 300 frames); mel filters vs librosa; frame decisions ≥ 99.9% vs the official Transformers output on a 60 s fixture. (Windows fixtures: `D:\jot-diar\fixtures\convo60.*`, not in the repo; ask the user.)
- **Accuracy:** a synthetic 2-voice TTS conversation with known turns, plus AMI ES2004a with its RTTM. Score the speaker per word at the word midpoint, best-mapped to the true labels.
- **No-regression:** transcripts of ≤ 5 min are byte-identical before and after. Long files (8–17 min) transcribe with clean seams.
