How We Built Baba Learn's Hebrew Pronunciation Engine

Why Couldn't Baba Learn Just Buy a Hebrew Pronunciation Scorer?

Baba Learn built its Hebrew pronunciation scoring engine by pairing gender-aware grapheme-to-phoneme generation through RenikudPlus with a custom wav2vec2 XLS-R CTC phoneme recognizer and alignment-free goodness-of-pronunciation scoring, replacing an earlier string-matching judge. The rebuild cut held-out sound error rate from 11.7% to 7.0% and raised pass accuracy on native test clips from 28 of 37 to 35 of 37.

No vendor sells a Hebrew pronunciation scorer as of October 2026. Microsoft's Azure Speech documentation lists Hebrew (he-IL) for speech-to-text but does not show Hebrew support for pronunciation assessment. Speechace and ELSA build for English learners first. That left a gap no API call could fill.

Hebrew text usually drops its vowel marks entirely, so a learner reading an unpointed sentence has to infer the vowels from context. Gender makes it harder. A man says "I want" as rotse; a woman says the same word as rotsa. "Yours" is shelcha to a man and shelach to a woman. Miss that ending and a learner hasn't just mispronounced a sound. They've addressed the wrong person.

A generic pronunciation scorer built for English or Spanish has no concept of any of this. It checks a recording against fixed text. Hebrew needs the scorer to know who is speaking, who they're speaking to, and which written form actually matches that pairing, before it can judge a single sound. That gap is why Maya, the voice tutor inside Baba Learn, runs a purpose-built engine instead of an existing API.

What Was Wrong With the First Pronunciation Judge?

Version one of the pronunciation judge, shipped in mid-September 2026, compared two text strings instead of listening for the truth. It converted the target sentence into expected sounds with an older Hebrew text-to-sounds model, converted the recording into sounds with a Whisper model fine-tuned to write Hebrew as IPA, aligned the two strings, and passed anything at an 80% match. The approach looked reasonable and produced wrong verdicts from both directions at once.

The expected-sound side had a gender problem baked in. The older text-to-sounds model read every feminine form as masculine, so a woman saying rotsa correctly got scored against a target built for rotse. It also mangled plain, ungendered words: boker, the Hebrew word for morning, came out of the model's own output as bakar. The target was broken before any recording reached it.

The heard-sound side had the opposite failure. Whisper is trained to write plausible Hebrew, so it tends to hear the word a native speaker would say, even when the learner said something else. A judge built on that instinct forgives real mistakes, because its ear is tuned to guess correctly rather than transcribe literally.

Free conversation turns had a third, quieter problem: they were scored against the learner's own transcript, which only proves two speech recognizers agreed with each other. It never proves the learner said anything correctly.

One small fix mattered more than it should have. The Whisper IPA model decoded short, single words to nothing until its decoding prompt was rebuilt by hand, using the timestamp token it had originally trained on. Without that fix, the judge produced no output for the shortest words, which are often the hardest ones to get right. The whole version looked like a working demo until it was checked against known answers, and it failed that check from both sides at once.

How Does Phoneme-Level Pronunciation Scoring Actually Work?

Phoneme-level pronunciation scoring compares each sound a learner was expected to make against what an acoustic model hears in the recording, not whole words or sentences. Three terms do the work. IPA, the International Phonetic Alphabet, writes sounds instead of letters. CTC, Connectionist Temporal Classification, is a training method that lets a speech model output a sequence of sounds without needing pre-marked boundaries in the audio. GOP, goodness of pronunciation, scores how confident the acoustic model is that a specific expected sound, and not some other sound, was actually produced at that point.

Older GOP methods needed forced alignment, a step that chops audio into fixed segments before scoring each one. That works against CTC-trained models, which produce confident spikes rather than evenly spaced segments. Researchers at the Norwegian University of Science and Technology, in a paper titled Segmentation-Free Goodness of Pronunciation, solve this by scoring every possible segmentation at once instead of committing to one in advance. They report that feature vectors built this way reach state-of-the-art phoneme-level results on the Speechocean762 benchmark. That self-aligned GOP approach, not a generic speech model, is what the Hebrew engine is built on.

Audio language models were tested and ruled out as judges. A 2025 study titled Exploring the Potential of Large Multimodal Models as Effective Alternatives for Pronunciation Assessment found that GPT-4o failed to return a complete result on 41.5% of clips in a pronunciation-assessment task. A separate preliminary study on GPT-4o's voice mode found audio LLMs scoring at chance level on phone-level tasks. An LLM can tell a learner their sentence sounded roughly fine. It can't reliably say which single sound was wrong, because it was never trained to listen at that resolution.

No off-the-shelf Hebrew phone recognizer existed to plug into this approach either. Hebrew wav2vec2 models output letters, not sounds, and Hebrew IPA tools are Whisper-based, the same normalize-toward-correct behavior that broke version one. Building the recognizer from scratch became the only path forward.

How Do You Turn a Hebrew Sentence Into the Right Expected Sounds?

Grapheme-to-phoneme conversion turns written text into the sounds it represents, the step that has to happen correctly before any audio gets scored. For Hebrew, that step can't read bare letters, because the same letters carry different vowels and different gender depending on who's speaking and who they're being spoken to.

The fix is to never guess. The app already knows the exact form shown on screen, vowel marks included, plus the speaker's gender and the listener's gender. It sends all of that to RenikudPlus, an open Hebrew grapheme-to-phoneme model, which treats the vowel marks and the two genders as hard constraints rather than suggestions. If the screen shows the feminine form, RenikudPlus returns the feminine sounds.

That matters because the alternative, guessing vowels and gender from bare letters the way the older model did, is exactly what turned boker into bakar and every feminine ending into a masculine one. RenikudPlus's published model card reports 99.3% correct on gender forms when the genders are known, against 59.9% for ReNikud, the earlier model it replaces.

Getting the target right before scoring a single sound is the least glamorous part of this build and the part that mattered most. A recognizer, no matter how accurate, can only be judged fairly against a target that's actually correct.

How Does the Engine Learn to Hear Hebrew Sounds and Pinpoint the Mistake?

Training a Hebrew sound listener meant building a recognizer that had never existed, then teaching it to disagree with a learner on purpose when the learner got something wrong. The build followed five steps.

1. Pick the base model. Start from wav2vec2 XLS-R 300M, a self-supervised speech model with 300 million parameters pretrained on 436,000 hours of unlabeled speech across 128 languages, according to Meta's published wav2vec2 XLS-R 300M model card on Hugging Face. Add a CTC head trained to output one of 27 Hebrew sounds at each step: five vowels and 22 consonants.
2. Build the training data. Run 104.6 hours of ivrit.ai crowd-transcribe-v5 podcast speech through the same pinned version of RenikudPlus the live judge uses, turning written Hebrew into the sounds a speaker of that transcript actually made.
3. Hold out real test data. Validate on two entire podcasts the model never trained on, and keep the dataset's own test podcasts as a final, untouched check.
4. Score with alignment-free GOP. For each sound the target says should be there, compare the likelihood of that sound against the same spot with every other sound swapped in, or deleted outright. The expected sound's share of that group is its score. Whichever sound wins the comparison is the sound the recognizer believes the learner actually made, which is what lets Maya say "you said b, try v" instead of just "wrong."
5. Confirm added sounds separately. When the recognizer's own decode shows an extra sound that wasn't in the target at all, that addition gets checked against likelihood before Maya reports it, so the system doesn't invent an added sound out of noise.

This is the recognizer no vendor had built for Hebrew. Training it, instead of calling an API, is what made sound-level feedback possible at all.

How Do Drills and Free Conversation Get Scored Differently?

Drills and free conversation get scored by different rules, because a drill has a known right answer and conversation doesn't. Treating them the same would either let drills pass garbage or make every conversational sentence feel like a pop quiz.

Before any scoring mode runs, Maya ignores variation native speakers produce anyway. Native Hebrew speakers themselves drop, add, or swap the glottal stop and the letter h depending on the word and the moment, and they commonly say the tsere vowel as "ey." None of that counts against a learner, because penalizing it would mean failing people for sounding native. Feedback instead concentrates on sounds English speakers actually get wrong: the throat r, the ch sound, b versus v, p versus f, k, and the word-final vowel that carries gender.

With that filter in place, three scoring modes apply:

1. Drills (repeat a word or line, fix the mistake) fail on any sound the recognizer is confident was wrong, defined as under 20% of its likelihood group with an alternative sound at least twice as likely. A word-final vowel, the one that can flip gender, fails at under 50%, a stricter bar than any other sound gets.
2. Sentences pass at 85% of sounds correct, unless the one wrong sound changes the meaning of the word, in which case the sentence fails regardless of the overall percentage.
3. Conversation turns only flag a clear miss on the hard sounds listed above and never fail the line outright, because the only available target in free conversation is the learner's own transcript, not a fixed correct answer.

That split keeps drills strict enough to teach and conversation loose enough to keep a learner talking.

How Was Gender Bias Found and Fixed in the Training Data?

Gender bias crept into the training data through a labeling default, not through any decision about Hebrew itself. RenikudPlus labeled training transcripts without knowing who was speaking to whom, so every ambiguous form defaulted toward masculine. Baba Learn's relabeling audit found the skew was stark: "lecha" (to you, masculine) was labeled 2,439 times across the training set, while "lach" (to you, feminine) was labeled just 82 times.

The first listener model learned that lean directly. The same audit tested it on 54 gender pairs across three voices and found it told only 42 apart, hearing a plain, clearly feminine "lach" as masculine in all three test voices, every time.

The fix didn't throw out the data. It generated every reading RenikudPlus could produce for each transcript, across every combination of speaker and listener gender, then let an early checkpoint of the model itself pick whichever generated reading best matched the actual audio. That checkpoint's pick became the new label. Running that process across the dataset changed 10,385 of 85,732 training labels, 12% of the total, in both directions, not just toward feminine, according to the same audit.

The retrained model, xlsr300m-v2, held sound error rate nearly flat at 6.95%, against 6.93% before the fix, while jumping gender-pair discrimination from 42 of 54 to 48 of 54. That model is what runs in production now.

The lesson generalizes past Hebrew. Any pipeline that auto-labels training data with a tool that has its own default inherits that default at scale. The fix isn't more data. It's finding exactly where the labeling tool guessed, and making it stop.

Did the Rebuild Actually Work? The Before-and-After Numbers

The rebuild worked, measured on September 27, 2026 against the same test clips the old judge had already scored. Every metric moved in the right direction, and the gap wasn't close.

The first four rows compare the old judge and the new one end to end. The last three compare the new recognizer with the old Whisper listener given the corrected target, which isolates hearing from the old target mistakes.

On the full 3,218-clip test set, the same September 27, 2026 evaluation recorded a 6.9% sound error rate for the new judge, the largest sample used in any single check during the rebuild.

The second row matters most. A judge that passes correct speech is only half the job; a judge that lets a deliberately wrong sound through is worse than no judge at all, because it tells a learner they're right when they aren't. The old judge caught the introduced mistake 7 times out of 37. The new judge caught it 35 times out of 37.

One finding settled the whole rebuild order: every failure the old judge produced on genuinely correct native speech traced back to its broken target, not to its ability to hear. That's why the work started with RenikudPlus and gender-aware phonemization before anyone touched the recognizer. Fixing the listener first would have meant training a better model against a target that was still wrong.

How Is the Engine Served in Production, and What's Still Open?

Baba Learn serves the pronunciation judge from a Python FastAPI service running on a Modal GPU, a T4 by default or an L4 when no T4 is free. Keeping a GPU warm around the clock would have cost about $430 a month, according to Baba Learn's own infrastructure measurements, so the service scales to zero instead, waking on a learner's first request with a ten-minute idle window before it sleeps again. A separate copy of the vowel-pointing model runs on CPU and is asked in parallel with the GPU. It also sleeps when nobody is learning, but it wakes far faster than the GPU, so vowel pointing comes back in seconds while the GPU is still booting.

Region mattered more than expected. With no region set, Modal placed the service in Jakarta, and a health check from US East took 292 ms. Pinned to us-east, the same check took 32 ms, per the team's latency logs. Every dependency is pinned to an exact version, not a range, and a test fails if a pin turns into one. When a third-party model the judge depended on disappeared from Hugging Face without warning, the team started mirroring the weights and training data to private backups every night. Recordings that come in silent or clipped return a neutral, low-confidence result instead of a fail, because the policy is simple: bad audio should never fail a learner.

Launch surfaced three more fixes. A compare endpoint now decodes a recording once and scores two to four gender candidates from that single decode, which cut warm median latency locally from 277 ms to 141 ms. A drill-start timing bug let a press cue bleed into the first fraction of a second of recording; native clips with that noise burst scored 33 to 57 until takes started getting cut to the word run that best fits the target, which brought those clips to 100 as of October 12, 2026. Drills missed three times in a row now get skipped instead of repeated forever, and conversation mode reports the specific sounds a learner missed, instead of a score that sat near 100 for almost everyone and told them nothing.

Not everything tried made it to production. Direct CTC candidate selection lost coverage. A normalization trick didn't improve separation. A wider search for added sounds cost latency without better verdicts. Bounded pilots of two other speech model families, W2V-BERT and Omnilingual with fresh heads, plus 600 more XLS-R training steps, didn't beat what was already live. Nothing shipped without beating the served model on the same clips first.

What's still open, stated plainly: stress isn't judged yet. Thresholds are cautious defaults, not numbers calibrated against native-speaker marks, because that calibration tooling is built and waiting on those marks. Every clip the models trained on is native speech, so how the system behaves on real learners, as opposed to clean podcast audio, is still being measured.

The lessons generalize past Hebrew. Build it yourself when no vendor sells the thing you need. Measure on clips with known right answers before trusting any model's output. Fix the target before touching the model. Never let a language model deliver a verdict it can't actually hear. Spend the budget on the learner's latency, not on a GPU sitting idle waiting for someone to talk.