Translating a one-line request into a shipped voice feature
The PM work a model can do
The request was one sentence: paste text and have it read back. The bug report was one word: harsh. Both were accurate and neither was actionable, so the engineering was the chain of translations in between. Here is what happened at each boundary between a description and the code, which crossings a model closed on its own, and the coherent plan that would have shipped the bug intact.
It arrived as one sentence: “I want to paste text and have it read back to me.”
I wrote that sentence myself, which makes this the easy case. The person who wanted the feature and the person who built it were the same person, with no account manager in the middle and no ticket to misread. The description still had to cross a chain of boundaries before anything could act on it, and it arrived corrupted at two of them.
As a spec that sentence does its job, because it names the user, the input and the outcome while leaving the implementation open. Everything that followed came from taking those limits seriously rather than reaching past them.
A read page went up the same morning: a text box, a play button, and audio streamed from a local speech model with about 200 milliseconds to first sound. Then the phone found two bugs the desktop never would have surfaced, which is the first boundary in the chain: a green test suite describes the machine it ran on, and you are always one step from acting on that description as though it covered the machine people use.
Safari split the auth token one way in a home-screen app and another way in a tab, and pulling AirPods off mid-sentence left the phone silent instead of falling back to the speaker. Both problems stayed invisible until a real device ran the real flow, because a simulator has no home screen and no headphones to pull off.
The simulator reported zero. The phone reported the truth. Your instruments only ever describe wherever you pointed them.
One visible
tap.
A visible handoff won because a description that fails out of sight leaves nobody anything to read, and the first live failure proved the point by lying about its own cause. Build the handoff you can still read.
Paste was still friction, so the next step was send: copy text anywhere on the phone, share it, and find it waiting in the read page.
Two flows were available. The share sheet could do the whole job without ever showing the app, or it could open the read page with the text already loaded and ask for one tap on Play.
I picked the visible version, because when a voice interface fails out of sight the person using it has no useful state left to inspect. One tap bought a place to see the payload and press Play. Design every handoff so that it still shows you state on the day a description turns out to be wrong, because that is the day you will need something to read.
iOS sets the boundary here: a third party cannot join the Copy and Look Up menu, so the share sheet is the only door available. The shortcut posts to a small queue API with a token, then opens the queued item in the read page.
Then the first live runs failed with “network connection was lost,” while the network was working fine. A bad token made the server close the socket instead of answering 401, and iOS translated that closed socket into a network error. That is a description crossing two boundaries and arriving as a confident lie, and two theories died on real hardware before the log exposed the actual fault.
That lane closed with 20 of 20 fetch tests and 24 of 24 queue tests passing.
Search the voices
you already have.
Warm, low and unhurried is a description, 0.56 times realtime is a number, and the number closed the path in an afternoon without anyone having to argue about taste. Numbers close arguments that adjectives keep open.
I wanted a narrator that felt warm, low, and unhurried. Somewhere in the Matt Damon neighborhood.
Voice cloning looked obvious, so five cloning engines entered the same benchmark. The fastest produced audio at 0.56 times realtime, meaning a one-second sentence takes almost two seconds to make. The production engine starts speaking in about 200 milliseconds.
That latency budget ended the cloning path, and the decision went into the record so nobody would keep reopening it from taste alone. When a number kills the approach you wanted, write the number down, because a call recorded as a preference gets reopened every month and a call recorded as a measurement does not.
The production model cannot clone, but it does ship 54 stock voices that can be blended with weights. So the question turned into a search: which blend lands closest to the target voice? A closed path hands you a smaller question.
Score the grid.
Keep the ear.
A metric translated taste into a ranking in about seven minutes, and it saturated well before it earned the right to translate that ranking back into a preference.
A speaker-verification model turned each candidate into a cosine-similarity score against reference clips of the target voice. The ceiling was 0.9522, measured from the reference clips against each other, while the default voice sat far below at 0.3860.
A grid searched 286 blend combinations in 425.5 seconds. Its leading blend used am_adam at 0.4, am_onyx at 0.4, and am_echo at 0.2. It scored 0.9462, close to the ceiling of what this metric could distinguish.
My own embarrassing number was elsewhere. Before the search I had guessed three ingredients from voice names and descriptions, and as single voices michael ranked sixth, fenrir eighth, and puck seventh out of nine.
My ear built a shortlist that the metric rejected, so the leading blend shipped as the default with a dropdown that keeps listener choice in the interface. The metric can tell you which candidates to audition and nothing about which one you should keep.
That boundary matters, because this similarity metric saturates near a same-speaker threshold of about 0.86, which makes a score near the ceiling a strong search signal and nothing more. If you hand a metric the job of ranking, keep the job of preferring somewhere a person can still reach it.
The adjective almost
cost a week.
“Harsh” described the symptom accurately and named the wrong mechanism, and the EQ plan built from it would have compiled, passed its own tests and left the real fault running underneath.
Then I listened to the voice read a real article. It sounded harsh. Rough around the edges.
I wrote a complete handoff from that impression: de-ess it, soften the highs, add light compression. The plan was coherent, and its entire diagnosis came from one adjective.
Instead, the next session started with measurement. The spectrum ruled out a sibilance spike and exposed a hard limiter running at six times gain, pinning 6.5% of all audio blocks flat.
Crest factor measures the distance between the peaks of a signal and its average level, which is another way of asking how much room the loud parts have left to move. The source started at 20.8 dB. The broken chain had crushed it to 11.1 dB, and those missing 9.7 dB are what an ear reports as harshness.
The harshness came from a gain stage nobody had listened to alone. Your ear found it. Your ear could not name it.
Build what the signal
can support.
What shipped answered the measured fault rather than the described one, and it kept its processing cost inside the latency budget that was already there.
What replaced it is a proper mastering chain: a high-pass filter, a low-mid cut, a safety de-esser, gentle compression, loudness normalization, and a lookahead limiter that catches peaks without flattening them.
There is a reverb stage too. It is built and switched off because audiobooks are dry, and this feature is an audiobook of whatever gets pasted into it.
Pinned blocks fell from 6.5% to zero, and crest factor recovered from 11.1 dB to 14.7 dB. The whole chain adds about 7 milliseconds per sentence, which fits inside a roughly 200 millisecond first-audio budget with room to spare.
The original de-ess plan could have shipped, compiled, and passed every test it wrote for itself while the limiter kept clipping underneath all of it. One measurement separated those two outcomes, and it is the cheapest thing you will do all week.
and wrong. Coherence is what a description buys you. The de-essing plan hung together perfectly, specified its own tests, and would have passed every one of them with the limiter still clipping underneath.
One build day,
one pair of ears.
The numbers above came from a single feature on a single machine, and four of them would move if you ran the same day differently.
Those similarity scores rank candidates against my reference clips of one target voice, so 0.9462 means close to those clips, which is a narrower claim than close to a voice you would pick for yourself. The ceiling of 0.9522 was measured the same way, leaving both numbers relative to a reference set of my own choosing.
That 286-combination grid searched three-voice blends at fixed weight steps, so it never tested the four-voice blends or the finer weights that might score higher. I stopped because the leading blend was already inside the metric's saturation band, not because the space was exhausted.
The 6.5% pinned-block figure and both crest-factor readings come from one article read by one voice, and denser source material would pin more blocks while sparser material would pin fewer. The direction of the fix holds either way, but treat the exact percentages as this sample rather than a specification.
Both the ~7 millisecond chain cost and the ~200 millisecond first-audio budget were measured on my hardware with nothing else competing for it. On a loaded machine both numbers grow, so the headroom that looks comfortable here will be tighter on yours.
Which translations a model can close.
Every boundary in this build got closed by whichever party could produce something checkable. The benchmark closed the cloning question, because 0.56 times realtime against a 200 millisecond budget is not a matter of opinion. The grid closed the blend question far enough to audition, the spectrum closed the harshness question in a few minutes, and the suites closed the queue at 20 of 20 and 24 of 24. None of those crossings needed me.
Two did. The share sheet could have done the whole job invisibly, and the visible tap stayed because a person debugging a voice interface needs state to read at the moment it fails. The dropdown stayed in the interface because a metric that saturates near 0.86 can rank candidates and cannot prefer one.
Both of those were my judgment, and they did not both hold. The dropdown call was right, while the plan I wrote from one adjective had the wrong diagnosis, the wrong target and the wrong fix. Every incentive in the room pointed at building it anyway, because the word mapped to a familiar tool and the tool could be built that afternoon.
Measurement felt like delay, and delay felt like failure. That is how a voice feature collects a de-esser, a warmth slider, and an EQ preset while a limiter clips 6.5% of every sentence underneath all three.
If you are holding a coherent plan that came out of a description rather than a number, that is the plan to check first. Measure what you can, then look hard at who decided the rest.
The person who asked for this feature never learned what crest factor is, and the question of which voice they actually prefer is still open in the dropdown. They stopped hearing the fault, which is the only report that ever comes back across the last boundary.
They stopped hearing the fault. That is the whole report.