chatterbox-tts/docs/superpowers/plans/2026-08-08-chatterbox-tts.md

Chatterbox TTS Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Deploy an OpenAI-compatible Chatterbox text-to-speech service at chatterbox.dimensionlab.net.

Architecture: A FastAPI application on loopback port 8881 wraps the CUDA Chatterbox model and exposes the standard speech endpoint. The application confines reference audio to its voices/ directory, serializes GPU synthesis, and unloads the cached model after exactly five idle minutes. A user systemd service supervises it, and Caddy exposes the public hostname.

Tech Stack: Python 3, FastAPI, Uvicorn, PyTorch, torchaudio, chatterbox-tts, systemd user units, Caddy.

Global Constraints

• Do not change the Kokoro service or its Caddy route.
• Bind Uvicorn to 127.0.0.1:8881 only.
• Use the English ResembleAI/chatterbox model on CUDA.
• Accept reference audio only below /home/vince/ai/apps/chatterbox-tts/voices.
• Set model idle unload to exactly 300 seconds.
• Permit at most one in-flight model synthesis request.
• Limit input to 2,000 characters.
• Support WAV responses in the first release.

---

Task 1: Create and test the API module

Files:

• Create: /home/vince/ai/apps/chatterbox-tts/app.py
• Create: /home/vince/ai/apps/chatterbox-tts/tests/test_app.py
• Create: /home/vince/ai/apps/chatterbox-tts/requirements.txt
• Create: /home/vince/ai/apps/chatterbox-tts/voices/.gitkeep

Interfaces:

• Produces: app: FastAPI with GET /health, GET /v1/models, and POST /v1/audio/speech.

• Produces: resolve_voice_path(voice: str) -> Path and SpeechRequest request validation.

• Step 1: Write failing API tests

def test_models_and_health(client):
    assert client.get('/health').json() == {'status': 'ok'}
    assert client.get('/v1/models').json()['data'][0]['id'] == 'chatterbox'

def test_voice_must_stay_under_voice_directory(client):
    response = client.post('/v1/audio/speech', json={
        'input': 'Hello', 'voice': '../secret.wav', 'response_format': 'wav'
    })
    assert response.status_code == 400

def test_input_is_limited(client):
    response = client.post('/v1/audio/speech', json={
        'input': 'x' * 2001, 'voice': 'reference.wav', 'response_format': 'wav'
    })
    assert response.status_code == 422

• Step 2: Run the tests to confirm they fail before implementation

Run: cd /home/vince/ai/apps/chatterbox-tts && .venv/bin/python -m pytest tests/test_app.py -v

Expected: FAIL because app.py and the app module do not exist.

• Step 3: Implement the API and GPU model manager

Implement a ModelManager protected by threading.Lock. It must lazily call ChatterboxTTS.from_pretrained(device='cuda'), update last_request_time at synthesis start, and have a daemon that clears the model and calls torch.cuda.empty_cache() when idle for 300 seconds. Guard the full inference call with the same lock. Resolve voice with Path(VOICES_DIR, voice).resolve() and reject it unless relative_to(VOICES_DIR.resolve()) succeeds. Use torchaudio.save to write the generated tensor to an in-memory WAV buffer.

@app.post('/v1/audio/speech')
def speech(request: SpeechRequest) -> Response:
    reference = resolve_voice_path(request.voice)
    wav, sample_rate = model_manager.generate(request.input, reference)
    buffer = io.BytesIO()
    torchaudio.save(buffer, wav.cpu(), sample_rate, format='wav')
    return Response(buffer.getvalue(), media_type='audio/wav')

• Step 4: Run the focused tests

Run: cd /home/vince/ai/apps/chatterbox-tts && .venv/bin/python -m pytest tests/test_app.py -v

Expected: PASS without loading the CUDA model for health, model-list, length, or invalid-path cases.

Task 2: Install isolated dependencies and verify real synthesis

Files:

• Modify: /home/vince/ai/apps/chatterbox-tts/requirements.txt
• Create: /home/vince/ai/apps/chatterbox-tts/tests/test_synthesis.py

Interfaces:

• Consumes: app.model_manager.generate(text, voice_path).

• Produces: a WAV-producing Chatterbox runtime in .venv.

• Step 1: Write the failing real-synthesis test

def test_generate_returns_audio_tensor_and_sample_rate(reference_wav):
    wav, sample_rate = model_manager.generate('A short Chatterbox smoke test.', reference_wav)
    assert wav.numel() > 0
    assert sample_rate > 0

• Step 2: Install the project environment

Run: cd /home/vince/ai/apps/chatterbox-tts && python3 -m venv .venv && .venv/bin/pip install --upgrade pip && .venv/bin/pip install -r requirements.txt

Expected: chatterbox-tts, CUDA-compatible PyTorch/torchaudio, FastAPI, Uvicorn, Pydantic, and pytest install into .venv only.

• Step 3: Add a short, consented reference WAV to voices/

Use one existing, locally owned test WAV as voices/reference.wav; do not copy any credentials or unrelated files.

• Step 4: Run the real-synthesis test

Run: cd /home/vince/ai/apps/chatterbox-tts && .venv/bin/python -m pytest tests/test_synthesis.py -v -s

Expected: PASS and the CUDA model is released after the test process exits.

Task 3: Supervise and publish the service

Files:

• Create: /home/vince/.config/systemd/user/chatterbox-tts.service
• Create: /etc/caddy/conf.d/chatterbox.caddy

Interfaces:

• Consumes: app:app from app.py.

• Produces: loopback service on port 8881 and HTTPS public hostname.

• Step 1: Create the systemd user unit

Use WorkingDirectory=/home/vince/ai/apps/chatterbox-tts and:

ExecStart=/home/vince/ai/apps/chatterbox-tts/.venv/bin/uvicorn app:app --host 127.0.0.1 --port 8881
Restart=on-failure
RestartSec=5

• Step 2: Create the Caddy route

chatterbox.dimensionlab.net {
    import cloudflare_tls
    reverse_proxy 127.0.0.1:8881
}

• Step 3: Validate configuration and activate services

Run: systemctl --user daemon-reload && systemctl --user enable --now chatterbox-tts.service && /usr/local/bin/caddy-cloudflare validate --config /etc/caddy/Caddyfile && sudo systemctl reload caddy

Expected: the user unit is active, Caddy validates, and the public route is loaded.

• Step 4: Verify the full public API path

Run:

curl -fsS https://chatterbox.dimensionlab.net/health
curl -fsS https://chatterbox.dimensionlab.net/v1/models
curl -fsS -o /tmp/chatterbox-smoke.wav \
  -H 'Content-Type: application/json' \
  -d '{"model":"chatterbox","input":"Service smoke test.","voice":"reference.wav","response_format":"wav"}' \
  https://chatterbox.dimensionlab.net/v1/audio/speech
file /tmp/chatterbox-smoke.wav

Expected: health and models return JSON, speech returns an RIFF/WAVE file, and no listener is exposed on a non-loopback interface for port 8881.