Your Voice Input Hallucinates on Silence. Here Is the Gate That Stops It.

A tester on a conversational agent with voice input opened the microphone, said nothing, and stopped the recording. A complete, fluent sentence appeared in the composer. It was not a garbled fragment or an obvious error. It read like something the user would plausibly have said, in the product's own register, using the product's own vocabulary.

That is the failure mode worth writing about. Everyone knows transcription models mishear words. Fewer teams plan for the case where the model invents a sentence out of room noise and hands it to a user who is about to press send.

This is what causes it, the design we shipped against it, the parameters that matter, and the two things we got wrong before it was right.

Why Silence Produces a Sentence

A speech-to-text model is a decoder. Given audio, it produces the most likely text. There is no branch in that architecture where the model concludes that nothing was said and returns an empty string. When the acoustic evidence is empty, the decoder still has to emit something, so it falls back to what it knows about likely text. You get a well-formed sentence with no source.

The part that surprised us was how our own accuracy work made this worse.

The product had shipped voice input weeks earlier with two accuracy mechanisms. First, the transcription call carries a prompt loaded with the domain glossary (the product's framework names, its coined phrases, the branded terms that generic transcription mangles) plus a slice of recent conversation. Second, a small chat model runs a strict correction pass afterward that fixes misrecognized terms without paraphrasing. Both work. Branded terms that came back phonetically wrong now decode correctly.

Both also steer the hallucination. Give a decoder no acoustic evidence and a prompt full of your domain vocabulary, and the sentence it invents is written in that vocabulary. The bias that fixed our accuracy problem made our silence problem convincing. QA logged it as fabricated in-domain phrases appearing from silent clips, and the root cause note in the ticket was blunt about it: no silence guard, and the bias prompt steers the hallucination.

If you have added prompt biasing to a transcription pipeline, you have this problem in a more dangerous form than a team that has not.

What We Considered and Did Not Build

Three options came up before the one we shipped.

A blocklist of known hallucination strings. Whisper-family models have famous stock fabrications, and you can string-match them out. We dropped it. It fixes the sentences you have already seen, never the next one, and it is the kind of guard that quietly rots as the model changes underneath you. It also does nothing about the real problem, which is that a clip with no speech in it reached a model at all.

Client-side detection in the browser. Attractive, because the cheapest API call is the one you never make. Resolved as server-only for this change. The browser is not a trustworthy place to enforce a content rule, capability varies across Chrome and iOS Safari, and we would have ended up maintaining the same decision in two places with two sets of numbers. The server already had the audio, and the decision belonged in one place.

Tuning first, shipping later. We had thresholds from a small proof of concept, thirteen clips, not a corpus. The temptation is to hold the fix until the numbers are properly tuned. We shipped the provisional numbers and made every gate decision log its inputs from day one instead, so tuning could happen against real traffic rather than against thirteen clips. That decision turned out to matter more than any threshold value, for reasons covered later.

The Design: Two Layers, Because One Cannot Cover It

The shipped design puts two independent checks between a recording and the composer.

Layer one is a voice activity gate that runs before the transcription API is ever called. The server decodes the browser's clip to a 16kHz mono waveform, scores it with a Silero VAD model, and rejects clips with no speech activity. Silence, room noise, breathing, and keyboard clatter never reach the transcription model, which also means they never cost an API call.

Layer two is a confidence backstop on the transcription call itself. A voice activity detector answers "is a human vocalizing here", not "is this intelligible speech". Voiced mumbling clears it on energy alone. So after a passing call, the service reads the average per-token logprob the transcription API already returned and rejects results below a floor. This costs nothing extra: switching response_format from text to json and adding include=["logprobs"] rides on the same request.

Both layers reject the same way, and this is the single most important design decision in the whole change:

class TranscriptionOutcome(BaseModel):
    text: str = ""
    raw_text: str | None = None
    corrected: bool = False
    status: Literal["ok", "no_speech", "clip_too_long"] = "ok"

no_speech is a 200, not an exception. "Nothing was said" is a correct, well-formed answer to "what is in this clip", not a failure. Modeling it as an error would have pushed it down the same path as a timeout or a bad key, and the user would have seen "something went wrong" for a situation where nothing went wrong. The frontend gets a distinct state and distinct copy ("Nothing heard, check your microphone") that auto-resets to idle so a retry is immediate.

One path stays a loud error. If a clip passes both the VAD gate and the model returns empty text anyway, that is a genuine model failure and it raises. We deliberately did not collapse it into no_speech, because that would hide a real defect inside the ordinary case.

Parameters, and What Each One Actually Does

The gate is an AND over three signals from the VAD, plus the logprob floor after it.

def passes_gate(result: VadResult, settings: Settings) -> bool:
    return (
        result.has_segment
        and result.speech_frame_ratio >= settings.vad_speech_ratio_threshold
        and result.max_prob >= settings.vad_max_prob_threshold
    )

has_segment runs Silero's own trigger and hangover state machine rather than a raw threshold count, so a brief pause does not split one utterance into two and probability chattering at the boundary does not register as a segment. We ported the boolean answer out of the reference implementation and verified it chunk for chunk against the original across every proof of concept clip before trusting it. Segment padding and the maximum-speech-duration split were left out on purpose: padding only shifts reported timestamps, which nothing here reads, and the split never fires for our input.

Every one of these numbers ships documented as provisional. That is not hedging, it is the actual state: they came from thirteen clips, and the code comments say so next to the values.

Four Implementation Details Worth Copying

Vendor the ONNX model, skip the pip package. The silero-vad package pulls torch and torchaudio as unconditional install dependencies even when only the ONNX backend runs at inference time. That is hundreds of megabytes of deploy weight for a gate that should be nearly free. We vendored the MIT-licensed 16kHz ONNX export (1.3MB) and wrote a numpy inference wrapper against onnxruntime directly, checked bit for bit against the package's own model call before it was trusted. No torch anywhere in the dependency tree.

The session is shared, the state is not. The model is recurrent: each 512-sample chunk carries a small hidden state into the next. The InferenceSession holds the weights, loads once at startup, and is safe to call concurrently. The state and context arrays are created fresh inside every evaluation call. Share those by accident and two concurrent requests corrupt each other's gate decisions, which is the sort of bug that shows up as unreproducible false rejects in production.

Keep the CPU work off the event loop. Decoding and inference are synchronous CPU work. The deploy tier runs a single uvicorn worker, so calling them directly would block that worker for the length of every gate check, stalling every other in-flight request including live chat token streams. Both go through asyncio.to_thread.

A byte cap on the upload does not bound decoded memory. This one came out of review and it is the trap most likely to bite someone reading this. The endpoint capped uploads at 25MB, matching the transcription API's own limit, which felt like enough. It is not: a low-bitrate clip expands to many times its compressed size as 16kHz float32 PCM, and a single request could push a 512MB box into an out-of-memory kill. The fix is a duration cap enforced during decode, raising as soon as the running total crosses it, before the rest of the container is pulled into memory.

def append(resampled: av.AudioFrame) -> None:
    nonlocal total_samples
    chunk = resampled.to_ndarray()[0]
    total_samples += len(chunk)
    if total_samples > max_samples:
        raise ClipTooLongError(
            f"decoded audio exceeded the {max_duration_s}s cap",
            declared_duration_s=_declared_duration_s(container),
        )
    chunks.append(chunk)

The declared_duration_s on that exception is worth a sentence of its own. Because the decode stops early, it can never observe how long the clip actually was, which is exactly the number you need to choose the cap. So the error carries the container header's own duration claim, read without decoding anything, and logs it. The mechanism that enforces the limit also reports the data needed to set the limit.

How We Tested It

Synthetic audio proves one direction and not the other. Numpy signals encoded through PyAV in memory (nothing recorded, nothing committed to the repo) cover the decode round trip for both browser containers, the gate math, the concurrency behavior of the shared session, and the near-cap and over-cap boundary. Silence reliably scores as silence, so the always-reject direction tests cleanly.

The opposite direction does not work synthetically. No synthesized signal reliably fools a neural VAD into scoring as real speech, so "real speech is not falsely rejected" can only be proven with real audio, on real devices, through the real browser codecs: Chrome recording webm/opus and iOS Safari recording mp4/aac, across silence, background noise, and speech. That was a manual pass, and it is the one surface the automated suite cannot replace.

The most useful test in the suite is a pair. The proof of concept produced a clip of voiced mumbling that measured a speech frame ratio of 0.438 and a peak probability of 0.996, which sails through the VAD gate, and an average logprob of -2.562, which the backstop rejects. Both numbers are pinned as regression tests: one asserting the gate cannot catch it, one asserting the backstop does. Together they are executable proof that the two-layer design is not redundant. If someone later decides the backstop is unnecessary, a test fails and tells them why it exists.

Final state at merge: 354 backend tests passing, 68 frontend tests, a live run against the real transcription API confirming a silent clip returns no_speech end to end, and a real four-minute recording returning clip_too_long in 0.2 seconds, which is itself the proof that rejection happens mid-decode rather than after a full expansion.

The Two Things We Got Wrong

Neither of these was caught by the test suite. Both were caught by using the feature.

The gate ate short answers. The proof of concept had been validated against full sentences only, never single words, so vad_min_speech_duration_ms at 250 and vad_max_prob_threshold at 0.98 were both tighter than a real one-word utterance can clear. In a conversational product, "yes" and "no" are among the most common things a user says. We dropped them to 50 and 0.95 based on logged gate decisions from real speech.

That fixed about a third of the failures. The rest were quiet speech that never registered as speech at all, scoring under even the base vad_threshold of 0.5. Loosening the peak-confidence bar does nothing for that group. The only lever that would is vad_threshold itself, which trades quiet-speech sensitivity directly against false-accepting background noise as speech, which is the exact failure this whole change exists to prevent. We left it alone and wrote the trade-off down rather than quietly tuning toward the bug we had just fixed.

Raising the duration cap broke the ratio. Review pushed the cap from 120 to 300 seconds on product grounds (a user thinking before answering an open-ended question is the normal case, not an edge case). That raise mechanically broke a threshold nobody was looking at, because speech_frame_ratio is speech-scored frames over the whole clip, not over the detected segment. A 40-second answer inside a 120-second clip caps around 0.33. The same answer inside a 300-second clip caps around 0.13, no matter how clearly the person spoke. We lowered the ratio floor from 0.05 to 0.01 as a provisional unblock and named the real fix (score the ratio over the detected segment, not the clip) as a follow-up needing production data.

The lesson generalizes past voice. A threshold defined as a ratio over a window is coupled to that window's size, and changing the window silently changes the threshold's meaning. If you cannot say what a parameter means when its denominator doubles, it is not a tuned value.

The Bug That Made the Whole Thing Unmeasurable

The service was written to log seven fields on every voice turn: speech ratio, peak probability, segment flag, duration, gate result, average logprob, and final status. The docstring stated plainly that this line is what post-launch threshold tuning would run against.

None of it reached the log. This is what a successful transcription printed:

2026-07-15 09:51:51,027 INFO src.services.transcription voice gate decision

Python's logging.basicConfig renders only the fields its format string names. Everything passed through extra={} was dropped, silently, everywhere in the application: about 140 call sites across 43 files. The fields existed, the calls were correct, and the output was empty.

We shipped provisional thresholds on the explicit promise that production traffic would tune them. That promise was worthless while the tuning data went nowhere. The fix was one Formatter subclass that appends a record's extra fields to the line, wired as the single root logger configuration, which fixed every other call site at the same time. Same clip, after:

voice gate decision vad_speech_ratio=0.0 vad_max_prob=0.0089 vad_has_segment=False
vad_duration_s=2.0 gate_pass=False avg_logprob=None final_status='no_speech' session_id=None

If you ship deliberately provisional numbers, the observability that justifies shipping them is part of the feature, not a follow-up. Verify it prints. We only found this because a review pasted actual log output as evidence instead of reading the logging code and assuming it worked.

The Other Silent Failure: Rejecting Well

A gate that rejects correctly and says nothing is still a bug. The first version had one.

Recording a normal turn that got rejected produced nothing at all. No text, no message, no sign of what happened. The work was simply gone. Four things stacked up to cause it: there was no client-side recording limit, so the clip was only measured after the user had finished talking; the rejection copy lived in an aria-label and nowhere on screen, which does not render visually and does not reliably announce to a screen reader either; the two rejection reasons rendered identically, so "you recorded silence" and "you recorded two minutes of real speech that ran past the cap" produced the same pixels; and the cap counted recording time rather than speech time, so thinking before answering burned the budget and took the real answer down with it.

What shipped instead: the recorder stops itself at the cap with a live countdown, so a stop is something the user sees coming rather than a loss discovered afterward. A capped recording transcribes whatever it captured and says the text may be incomplete, rather than discarding it. The two rejection states render as real text next to the composer with distinct icons. And the cap value is served from the backend through a small config endpoint, so the frontend cannot hold a stale copy of a number the backend owns. That last one closed a specific trap: the original code hardcoded "keep it under two minutes" in the copy while the real bound was a backend setting, so the moment we raised the cap the interface would have started lying, and fixing it would have required the redeploy that making it a setting was supposed to avoid.

What To Take From This

If you are running speech-to-text in production, three things are worth taking directly.

First, decide what an empty clip means before a user finds out for you. The model will never tell you nothing was said. Something in your pipeline has to, and it should be a successful outcome with its own state, not an error.

Second, prompt biasing raises the stakes. Every term you add to improve accuracy is also a term the model can reach for when it has nothing to transcribe. The better your glossary, the more convincing your hallucinations.

Third, one detector is not enough, and know why. A voice activity model answers a different question than "is this intelligible speech", and the gap between those two questions is exactly where mumbling lives. The confidence signal that closes it is already in the API response you are paying for.

Everything else is tuning, and tuning is a data problem. Ship the guard with numbers you can defend as provisional, log every decision it makes, and be honest in the code comments about which values are measured and which are placeholders waiting for traffic.

Vindler builds production AI systems for teams that need them to work after launch, not just in the demo. If you are running voice, RAG, or agent infrastructure and hitting failures like this one, book a call.