How AI pronunciation scoring works (a plain-English tour)

You press record, read a sentence, and two seconds later an app tells you your /θ/ in think slipped toward /s/. What happened in those two seconds? This post walks through the whole pipeline — no math degree required — and explains why the architecture of the scoring engine decides whether the feedback you get is trustworthy or flattering.

We build sayit, so this is also a description of how our engine works. But the pipeline below is the honest shape of any serious pronunciation assessment AI, and knowing it will help you evaluate every app on the market.

Step 0: what the machine actually receives

Your microphone does not capture words. It captures a pressure wave — thousands of amplitude samples per second. The first job of any speech system is to turn that wave into something a neural network can read: a sequence of short overlapping slices (a few tens of milliseconds each), each summarized as a small grid of numbers describing which frequencies carry energy at that instant.

That is the raw material. Everything a pronunciation scorer knows about you, it knows from those slices.

Step 1: a neural acoustic model guesses the sound in every slice

Next, a neural acoustic model — in sayit's case a wav2vec2-family network trained on huge amounts of speech — looks at each slice in context and outputs a probability for every phoneme of English: "this instant is 92% /s/, 5% /θ/, 1% /f/…". Engineers call these phoneme posteriors. Think of them as a running commentary: for every fraction of a second, the model's honest belief about which of the 44 sounds of English it is hearing.

Here is the detail that matters more than anything else in this post: sayit's acoustic model has no language model attached. It is never asked "what word is this?" — only "what sound is this?". A normal speech-to-text system (the kind behind dictation and AI conversation apps) adds a language model that pulls ambiguous audio toward likely words, which is exactly what you want for transcription and exactly what you do not want for assessment, because it silently repairs your mistakes. That failure mode is big enough that we wrote a whole post on the auto-correct problem.

Step 2: alignment — matching what you said to what you should have said

The app knows the sentence you were trying to read, so it also knows the target phoneme sequence. "I think so" is, in IPA, /aɪ θ ɪ ŋ k s oʊ/. The next stage stretches and matches the model's frame-by-frame beliefs against that target sequence — deciding which slices of your audio belong to the /θ/, which to the /ɪ/, and so on. This is called alignment, and it is how the system can point at one specific sound inside a fluent sentence instead of judging the sentence as a blob.

Alignment also catches structural errors: a phoneme you skipped entirely (many learners drop final consonants), a vowel you inserted (Spanish speakers often say "espeak" for "speak"), or a word you never attempted — which is where a completeness score comes from.

Step 3: per-phoneme goodness scores

Now the scoring itself is almost simple. For each target phoneme, the system asks: during the slices aligned to this sound, how much did the acoustic model believe it was hearing this sound — versus something else?

• If the frames aligned to your /θ/ are confidently /θ/, that phoneme scores high.
• If they look mostly like /s/, the score drops — and, crucially, the system knows what you said instead.

This family of measures descends from a classic research idea called goodness of pronunciation: score each sound by comparing the probability of the intended phoneme against the competition. It is the reason sayit can render a color-coded heatmap over every word of your sentence and, for each weak sound, show the target IPA next to the sound you actually produced.

That substitution information is the difference between a score and a diagnosis. "72%" tells you to try again. "Your /θ/ came out as /s/ — put your tongue tip between your teeth and let air flow" tells you what to do. It is why sayit shows IPA target-vs-actual plus an articulation tip instead of a bare percentage, and it is what makes drills like minimal pairs targeted instead of random.

Step 4: prosody — because pronunciation is more than phonemes

You can produce every phoneme correctly and still sound off, because English carries meaning in pitch, timing and emphasis. So alongside the phoneme scores, the pipeline measures:

• Stress. English words have strong and weak syllables — PHOtograph vs phoTOGraphy — and stressing the wrong one damages intelligibility more than most single-sound errors. The system compares the loudness, length and pitch of your syllables against the expected pattern. (More in word stress and intonation.)
• Intonation. The pitch contour of the sentence: does your voice fall at the end of a statement, rise on a yes/no question, stay flat when it should move?
• Tempo and fluency. Speaking rate, pauses in unnatural places, hesitations and restarts. A long silence mid-phrase reads very differently from a pause at a comma.

sayit folds these into separate fluency, intonation and stress scores rather than mixing them into one number, because the fix for a stress problem (drill the syllable pattern) is nothing like the fix for a segmental one (change your tongue position).

Why the transcript-matching shortcut fails

Plenty of "AI pronunciation" features skip all of the above. They run ordinary speech-to-text on your recording and compare the transcript to the target sentence. Words match: green. Words differ: red.

The problem: the recognizer was trained to produce likely text. Say "I sink so" and a good transcription engine writes "I think so", because that is what English speakers usually mean — so the app awards a perfect score to the exact error you needed to catch. Transcript matching measures whether a very forgiving listener could guess your words, not how you pronounced them. And it can be wrong in the harsh direction too: an unusual-but-fine regional pronunciation can get transcribed as the wrong word and marked as an error — one source of the demoralizing false positives we discuss in accent-tolerant scoring.

A phoneme-level pipeline has the opposite virtues. It hears the raw sounds, so real errors surface; and because it scores contrasts rather than word matches, it can be deliberately tolerant of regional variation while enforcing the distinctions that change meaning (ship/sheep, rice/lice).

What this looks like in practice

Put the pipeline together and a single ten-second recording in sayit yields:

1. 1.A per-word, per-phoneme heatmap of the whole sentence.
2. 2.For each weak sound: target IPA, the sound you actually made, and a concrete articulation tip.
3. 3.Fluency, completeness, intonation and stress scores.
4. 4.Over time, analytics on which phonemes are systematically weak — which is what the Magic Wand coach uses to auto-build your personal drill plan across the practice exercises.

You can run this on 500+ leveled passages, AI-generated paragraphs, your own PDFs, or full IELTS-style speaking exams — the engine is the same everywhere. Individual words you are unsure about live in the word lookup hub.

The two-minute test for any app

You now know enough to audit any pronunciation app, including ours. Record a sentence with one deliberate, specific error — "I sink the answer is right." A phoneme-level engine will flag /θ/ → /s/ on that one word. A transcript-matcher will congratulate you.

Try it on sayit free — it runs in the browser, no install, no card. And if you are curious how the rest of the product wraps this engine, the features page has the full tour.