The Hidden Latency Challenge

In a voice agent, perceived responsiveness begins the moment the user stops speaking. If the system is slow to detect "the user is done" everything downstream (LLM, TTS, playback) feels laggy—even if those parts are fast.

What EoU Delay Actually Is

When I started digging into where latency was creeping in, I decided to begin right at the start of the pipeline — End-of-Utterance (EOU) detection.

LiveKit's EoU metric aggregates:

1. VAD silence detection (e.g., min_silence_duration)
2. STT transcription delay (time for final transcript to arrive, that will be sent to Turn detector)
3. Turn detector model inference (EoU classifier)

Key formula:
EOU Delay ≈ VAD silence + Turn-detector inference + STT finalization

---

Starting with the Turn Detector

LiveKit ships with its own turn detector models. They're solid, but not exactly lightweight. The English model averages around 150 ms (on my system) inference, while the multilingual one takes about 215 ms (on my system). Both of them also use the last six turns of conversation as input, which makes sense for context but adds overhead.

I wanted something faster, leaner, and more specialized for our use case. That's how I ended up building a custom SmolLM2-135M model, exported to ONNX and integrated directly with LiveKit.

Instead of looking at multiple turns, my model only focused on the current user utterance. It turned out that for EOU classification, this was more than enough. And the payoff was big: inference dropped to around 25–30 ms, a clear 5–6x improvement over LiveKit's defaults.

At this point, I felt I had cracked the problem. But when I looked at LiveKit's reported EOU delay, it still hovered in the range of 400–500 ms.

---

The Hidden Culprit: Transcription Delay

LiveKit's reported "EOU delay" isn't just about the turn detector. It actually bundles three things together:

1. VAD silence detection – basically waiting for a small pause in speech
2. Turn detector inference – the part I had optimized to ~25 ms
3. STT transcription delay – the time the speech-to-text provider takes to deliver the final transcript

The third piece was the real surprise. While VAD and my model were consistent (around 100 ms and 25–30 ms), the STT finalization was all over the place. Sometimes it was instant, other times it lagged by 400+ ms. And because LiveKit bundles that into the EOU metric, it completely masked the improvements I had made on the model side.

In short: the turn detector wasn't the main culprit — the STT provider was.

---

Lessons Along the Way

While building and integrating the custom model, I stumbled into a few interesting side notes:

1. Message handling quirks: LiveKit messages store content as arrays. Using the text_content property was the clean way to avoid mismatches.

2. Single-utterance vs. multi-turn: I initially thought ignoring history might hurt accuracy. But for detecting if a user has stopped speaking, focusing on the current utterance worked perfectly — and kept the model simple and fast.

3. Accuracy stayed strong: Despite being smaller and faster, the detector maintained 92% confidence in its predictions with 91.70% precision, 94.17% recall and 92.92% F1 score.

---

Key Technical Breakthroughs

1. Content Extraction Fix

LiveKit messages store content in arrays, which initially caused issues. I solved this by safely using the text_content property.

2. Single-Utterance Focus

While at first it felt like a limitation, training only on isolated utterances actually fit the EOU task perfectly and cut overhead.

3. Stable High Accuracy

Despite being smaller and faster, the model maintained 94–98% confidence accuracy on EOU detection.

---

Conclusion

The journey to optimize voice agent responsiveness revealed that the bottleneck wasn't where I initially expected. While building a faster turn detector model was valuable, the real latency culprit was STT transcription delays. This experience reinforced the importance of measuring the entire pipeline, not just individual components.

The custom SmolLM2-135M model now serves as a lean, efficient solution that maintains high accuracy while delivering 5-6x performance improvements over default LiveKit models.