The Nightmare of Local Speaker Diarization (And How We Dockerized It)
How we combined Faster-Whisper, VAD, and Speaker Diarization into a single zero-dependency Docker container for air-gapped enterprise environments.
OpenAI's Whisper is arguably the most significant open-source contribution to speech recognition. But if you’ve ever tried to build a production-grade meeting transcription pipeline, you know Whisper has a fatal flaw: It doesn't know who is speaking.
In the B2B SaaS world, a transcript without speaker tags is just a useless wall of text. You need Speaker Diarization.
In this post, I'll walk through the architectural nightmares of combining ASR (Automatic Speech Recognition) with Diarization, and how we ultimately solved it by unifying the pipeline into a single, air-gapped Enterprise Docker image.
The Architecture of "Who Spoke When"
To get a perfect transcript with speaker tags, you can't just run one AI model. You need a highly orchestrated pipeline:
VAD (Voice Activity Detection): Filters out silence and background noise to find actual speech segments.
Diarization (e.g., PyAnnote / CAM++): Extracts voice embeddings and clusters them (identifying SPK_1, SPK_2, etc.).
ASR (Faster-Whisper): Transcribes the actual words.
Alignment: Merging the timestamps from the Diarization model with the word-level timestamps from Whisper.
The Dependency Hell
If you try to build this locally, you will immediately run into "dependency hell".
Whisper relies on specific versions of
CTranslate2.Diarization models often require conflicting versions of
PyTorchortorchaudio.Running this on an NVIDIA GPU requires precise matching of CUDA toolkits and cuDNN libraries.
When our enterprise clients asked to run our transcription engine on their own AWS VPCs for privacy reasons, we knew we couldn't just hand them a requirements.txt and wish them luck.
The Solution: A Zero-Dependency Docker Container
To solve this, we pivoted to an immutable infrastructure approach. We packaged the entire pipeline—ASR, VAD, Diarization, and a FastAPI backend—into a single Docker image.
1. Model Caching at Build Time
Instead of downloading multi-gigabyte models at runtime (which fails in air-gapped environments without internet access), we inject the AI models directly into the Docker image layers during the build process:
# Pre-baking huggingface/modelscope models into the image
COPY models_cache/huggingface /root/.cache/huggingface
COPY models_cache/modelscope /root/.cache/modelscope
2. Cython IP Protection
Because we deploy this to enterprise servers, we needed to protect the proprietary alignment algorithms. We added a build step that uses Cython to compile all core Python logic into .so binary extensions, entirely removing the human-readable .py source code from the final image.
3. Unified REST API
The container exposes a simple /api/transcribe endpoint. The client uploads an audio file, and the internal Python engine handles the complex VAD -> Diarization -> Whisper -> Alignment pipeline natively on the GPU, returning clean JSON.
Why This Matters for B2B
For sectors like legal, medical, and finance, uploading internal meeting recordings to public cloud APIs is a massive security violation. By providing a self-hosted, Dockerized pipeline, companies get state-of-the-art AI transcription while keeping 100% of their data strictly inside their own firewall.
If you are building products for privacy-conscious industries, stop fighting with PyTorch dependencies. Check out how we structured our Enterprise Docker Edition at FreeAudioToText.
