> ## Documentation Index > Fetch the complete documentation index at: https://docs.sully.ai/llms.txt > Use this file to discover all available pages before exploring further. # Audio Transcription > Master audio capture with file upload and real-time WebSocket streaming ## Overview Sully.ai provides two approaches for converting patient conversations to text: | Approach | Best For | Latency | | ----------------------- | ---------------------------------------------------------- | -------------------------- | | **File Upload** | Pre-recorded audio, batch processing, large files | Async (seconds to minutes) | | **Real-time Streaming** | Live visits, immediate feedback, interactive transcription | Real-time (\~200ms) | Both approaches produce the same high-quality medical transcription output that can be passed to [note generation](/documentation/notes#2-generate-a-note). ## File Upload Upload pre-recorded audio files for asynchronous transcription. This approach is ideal for batch processing, large files, or when real-time feedback is not required. ### Supported Formats | Format | MIME Type | Extension | | ------ | ------------ | --------- | | WAV | `audio/wav` | `.wav` | | MP3 | `audio/mpeg` | `.mp3` | | FLAC | `audio/flac` | `.flac` | | OGG | `audio/ogg` | `.ogg` | | WebM | `audio/webm` | `.webm` | | MP4 | `audio/mp4` | `.mp4` | | M4A | `audio/mp4` | `.m4a` | | AAC | `audio/aac` | `.aac` | | Opus | `audio/opus` | `.opus` | Maximum file size: **100MB**. For larger files, consider splitting into segments or using real-time streaming. ### Upload and Poll File transcription is asynchronous. Submit your file, then poll for completion. ```typescript TypeScript SDK theme={null} import SullyAI from '@sullyai/sullyai'; import * as fs from 'fs'; const client = new SullyAI(); // 1. Upload audio file const transcription = await client.audio.transcriptions.create({ audio: fs.createReadStream('patient-visit.mp3'), }); console.log(`Transcription ID: ${transcription.transcriptionId}`); // 2. Poll until complete let result = await client.audio.transcriptions.retrieve( transcription.transcriptionId ); while (result.status === 'STATUS_PROCESSING') { await new Promise((resolve) => setTimeout(resolve, 2000)); result = await client.audio.transcriptions.retrieve( transcription.transcriptionId ); } if (result.status === 'STATUS_ERROR') { throw new Error('Transcription failed'); } console.log('Transcript:', result.payload?.transcription); ``` ```python Python SDK theme={null} import time from sullyai import SullyAI client = SullyAI() # 1. Upload audio file with open("patient-visit.mp3", "rb") as audio_file: transcription = client.audio.transcriptions.create(audio=audio_file) print(f"Transcription ID: {transcription.transcription_id}") # 2. Poll until complete while True: result = client.audio.transcriptions.retrieve(transcription.transcription_id) if result.status == "STATUS_SUCCEEDED": print(f"Transcript: {result.payload.transcription}") break elif result.status == "STATUS_ERROR": raise Exception("Transcription failed") time.sleep(2) ``` ```bash HTTP theme={null} # 1. Upload audio file curl -X POST "https://api.sully.ai/v2/audio/transcriptions" \ -H "X-API-Key: ${SULLY_API_KEY}" \ -H "X-Account-Id: ${SULLY_ACCOUNT_ID}" \ -F "audio=@./patient-visit.mp3" # Response: { "data": { "transcriptionId": "tr_abc123" } } # 2. Poll for completion curl -X GET "https://api.sully.ai/v2/audio/transcriptions/tr_abc123" \ -H "X-API-Key: ${SULLY_API_KEY}" \ -H "X-Account-Id: ${SULLY_ACCOUNT_ID}" # When complete: # { "data": { "status": "completed", "payload": { "transcription": "..." } } } ``` ### Dictation Formatting If you want prerecorded transcript output formatted for dictation workflows, add the optional `dictation` field and set it to `true`. If omitted, `dictation` defaults to `false`. ```typescript TypeScript theme={null} import { readFile } from 'fs/promises'; const audioBytes = await readFile('./patient-visit.mp3'); const formData = new FormData(); formData.append( 'audio', new File([audioBytes], 'patient-visit.mp3', { type: 'audio/mpeg' }) ); formData.append('dictation', 'true'); await fetch('https://api.sully.ai/v2/audio/transcriptions', { method: 'POST', headers: { 'X-API-Key': process.env.SULLY_API_KEY!, 'X-Account-Id': process.env.SULLY_ACCOUNT_ID!, }, body: formData, }); ``` ```python Python theme={null} import os import requests with open("patient-visit.mp3", "rb") as audio_file: response = requests.post( "https://api.sully.ai/v2/audio/transcriptions", headers={ "X-API-Key": os.environ["SULLY_API_KEY"], "X-Account-Id": os.environ["SULLY_ACCOUNT_ID"], }, files={ "audio": ("patient-visit.mp3", audio_file, "audio/mpeg"), }, data={ "dictation": "true", }, ) response.raise_for_status() ``` ```bash HTTP theme={null} curl -X POST "https://api.sully.ai/v2/audio/transcriptions" \ -H "X-API-Key: ${SULLY_API_KEY}" \ -H "X-Account-Id: ${SULLY_ACCOUNT_ID}" \ -F "audio=@./patient-visit.mp3" \ -F "dictation=true" ``` ### Status Lifecycle | Status | Description | | ------------ | ---------------------------------------------- | | `pending` | Request received, queued for processing | | `processing` | Actively being transcribed | | `completed` | Transcription ready in `payload.transcription` | | `failed` | An error occurred | For production applications, use [webhooks](/api-reference/events/webhooks) instead of polling to receive notifications when transcription completes. ## Real-time Streaming Stream audio in real-time during patient visits for immediate transcription feedback. This approach uses WebSockets to send audio chunks and receive transcription segments as they are processed. ### Connection Flow ``` 1. Get token POST /v1/audio/transcriptions/stream/token 2. Connect wss://api.sully.ai/v1/audio/transcriptions/stream?... 3. Wait for { "type": "status", "status": "connected" } 4. Send audio { "audio": "" } 5. Receive status, transcript, and error messages 6. Close ws.close() ``` ### Get a Streaming Token Before connecting to the WebSocket, obtain a short-lived token: ```typescript TypeScript theme={null} const tokenResponse = await fetch( 'https://api.sully.ai/v1/audio/transcriptions/stream/token', { method: 'POST', headers: { 'X-API-Key': process.env.SULLY_API_KEY!, 'X-Account-Id': process.env.SULLY_ACCOUNT_ID!, }, } ); const { data: { token } } = await tokenResponse.json(); ``` ```python Python theme={null} import requests import os response = requests.post( "https://api.sully.ai/v1/audio/transcriptions/stream/token", headers={ "X-API-Key": os.environ["SULLY_API_KEY"], "X-Account-Id": os.environ["SULLY_ACCOUNT_ID"], } ) token = response.json()["data"]["token"] ``` ```bash HTTP theme={null} curl -X POST "https://api.sully.ai/v1/audio/transcriptions/stream/token" \ -H "X-API-Key: ${SULLY_API_KEY}" \ -H "X-Account-Id: ${SULLY_ACCOUNT_ID}" # Response: { "data": { "token": "eyJ..." } } ``` ### WebSocket URL Connect to the streaming endpoint with your token and audio parameters: ``` wss://api.sully.ai/v1/audio/transcriptions/stream?sample_rate=16000&account_id={accountId}&api_token={token} ``` | Parameter | Required | Description | | ------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sample_rate` | No | Audio sample rate in Hz (e.g., `16000`, `44100`). If omitted, the streaming service currently defaults to `16000`. For raw, headerless audio, send the actual sample rate explicitly. | | `account_id` | Yes | Your Sully account ID | | `api_token` | Yes | Token from the stream token endpoint | | `language` | No | BCP47 language tag (e.g., `en`, `es`, `multi`) | | `dictation` | No | Set to `true` to request dictation-oriented transcript formatting | For raw, headerless audio, send both `encoding` and `sample_rate`. For containerized audio, omit `encoding`. If `sample_rate` is omitted, the current streaming service still defaults it to `16000`. ### Message Format **Stream ready:** ```json theme={null} { "type": "status", "status": "connected", "timestamp": "2026-04-22T14:32:07.123Z" } ``` **Sending audio:** ```json theme={null} { "audio": "" } ``` **Receiving transcription:** ```json theme={null} { "type": "transcript", "text": "The patient reports feeling tired", "isFinal": false, "is_final": false } ``` **Error message:** ```json theme={null} { "type": "error", "error": "An error occurred during transcription", "timestamp": "2026-04-22T14:32:12.123Z" } ``` * Wait for the `status: connected` message before sending audio * `text`: The transcribed text for the current segment * `is_final`: Canonical finality flag for the current segment * `isFinal`: Compatibility alias for `is_final` * `type: "error"` indicates a transcription problem. Some runtime errors are non-terminal, while other failures are followed by socket closure. ### Basic WebSocket Connection ```typescript TypeScript theme={null} // Get streaming token first (see above) const token = await getStreamingToken(); const accountId = process.env.SULLY_ACCOUNT_ID!; // Connect to WebSocket const ws = new WebSocket( `wss://api.sully.ai/v1/audio/transcriptions/stream?sample_rate=16000&account_id=${accountId}&api_token=${token}` ); // Track transcription segments const segments: string[] = []; let currentIndex = 0; let streamReady = false; ws.onopen = () => { console.log('Socket opened, waiting for stream readiness...'); }; ws.onmessage = (event) => { const data = JSON.parse(event.data); if (data.type === 'status') { streamReady = data.status === 'connected'; console.log(`Stream status: ${data.status}`); return; } if (data.type === 'error') { console.warn('Transcription stream error:', data.error); return; } if (data.type === 'transcript' && data.text) { segments[currentIndex] = data.text; const isFinal = data.is_final ?? data.isFinal ?? false; if (isFinal) { console.log(`Segment ${currentIndex}: ${data.text}`); currentIndex++; } } }; ws.onerror = (error) => { console.error('WebSocket error:', error); }; ws.onclose = (event) => { console.log(`Connection closed: ${event.code} ${event.reason}`); const fullTranscript = segments.join(' '); console.log('Full transcript:', fullTranscript); }; // Send audio data (from microphone, file, etc.) function sendAudio(audioBuffer: ArrayBuffer) { if (!streamReady) { console.warn('Stream not ready yet; wait for the connected status message.'); return; } const base64Audio = btoa( String.fromCharCode(...new Uint8Array(audioBuffer)) ); ws.send(JSON.stringify({ audio: base64Audio })); } ``` ```python Python theme={null} import asyncio import websockets import json import base64 import os async def stream_transcription(): # Get streaming token first (see above) token = await get_streaming_token() account_id = os.environ["SULLY_ACCOUNT_ID"] url = f"wss://api.sully.ai/v1/audio/transcriptions/stream?sample_rate=16000&account_id={account_id}&api_token={token}" segments = [] current_index = 0 stream_ready = False async with websockets.connect(url) as ws: print("Socket opened, waiting for stream readiness...") async def receive_messages(): nonlocal current_index, stream_ready async for message in ws: data = json.loads(message) if data.get("type") == "status": stream_ready = data.get("status") == "connected" print(f"Stream status: {data.get('status')}") continue if data.get("type") == "error": print(f"Transcription stream error: {data.get('error')}") continue if data.get("type") == "transcript" and "text" in data: # Extend segments list if needed while len(segments) <= current_index: segments.append("") segments[current_index] = data["text"] is_final = data.get("is_final", data.get("isFinal", False)) if is_final: print(f"Segment {current_index}: {data['text']}") current_index += 1 async def send_audio(audio_data: bytes): if not stream_ready: print("Stream not ready yet; wait for the connected status message.") return base64_audio = base64.b64encode(audio_data).decode("utf-8") await ws.send(json.dumps({"audio": base64_audio})) # Start receiving messages receive_task = asyncio.create_task(receive_messages()) # Send your audio data here # await send_audio(audio_chunk) # Wait for completion await receive_task full_transcript = " ".join(segments) print(f"Full transcript: {full_transcript}") asyncio.run(stream_transcription()) ``` ## Production Streaming Real-time audio streaming in production requires handling network interruptions, reconnection, and audio buffering. This section provides battle-tested patterns for reliable streaming. ### Key Challenges 1. **Network interruptions** - Mobile networks and WiFi can drop unexpectedly 2. **Token expiration** - Streaming tokens have limited validity 3. **Audio continuity** - Buffering audio during reconnection to prevent data loss 4. **State recovery** - Resuming transcription context after reconnection 5. **Error frames** - Some server error messages are non-terminal, while others precede disconnects ### Reconnection with Exponential Backoff Never reconnect immediately after a failure. Use exponential backoff with jitter to prevent thundering herd problems: ```typescript theme={null} interface BackoffConfig { baseDelayMs: number; maxDelayMs: number; maxAttempts: number; } function calculateBackoff( attempt: number, config: BackoffConfig ): number { const exponentialDelay = config.baseDelayMs * Math.pow(2, attempt); const cappedDelay = Math.min(exponentialDelay, config.maxDelayMs); // Add jitter: random value between 0-25% of delay const jitter = cappedDelay * Math.random() * 0.25; return cappedDelay + jitter; } ``` ### Production WebSocket Implementation The following implementation handles reconnection, audio buffering, and error recovery: This example reconnects on close events and connection failures. In your own implementation, surface `type: "error"` messages immediately, but do not assume every error frame is terminal. Some runtime errors are followed by later stream messages, while other failures are followed by socket closure. ```typescript TypeScript theme={null} import { EventEmitter } from 'events'; interface StreamConfig { accountId: string; apiKey: string; sampleRate: number; language?: string; maxReconnectAttempts?: number; baseReconnectDelayMs?: number; maxReconnectDelayMs?: number; audioBufferMaxSize?: number; } interface TranscriptionSegment { index: number; text: string; isFinal: boolean; } type ConnectionState = | 'disconnected' | 'connecting' | 'connected' | 'reconnecting'; class ProductionTranscriptionStream extends EventEmitter { private ws: WebSocket | null = null; private state: ConnectionState = 'disconnected'; private reconnectAttempt = 0; private audioBuffer: string[] = []; private segments: string[] = []; private currentSegmentIndex = 0; private token: string | null = null; private abortController: AbortController | null = null; private readonly config: Required; constructor(config: StreamConfig) { super(); this.config = { maxReconnectAttempts: 5, baseReconnectDelayMs: 1000, maxReconnectDelayMs: 30000, audioBufferMaxSize: 100, // Buffer up to 100 audio chunks language: 'en', ...config, }; } async connect(signal?: AbortSignal): Promise { if (signal?.aborted) { throw new Error('Connection aborted'); } this.abortController = new AbortController(); this.state = 'connecting'; this.emit('stateChange', this.state); try { // Get fresh token this.token = await this.fetchToken(); await this.establishConnection(); } catch (error) { this.state = 'disconnected'; this.emit('stateChange', this.state); throw error; } } private async fetchToken(): Promise { const response = await fetch( 'https://api.sully.ai/v1/audio/transcriptions/stream/token', { method: 'POST', headers: { 'X-API-Key': this.config.apiKey, 'X-Account-Id': this.config.accountId, }, signal: this.abortController?.signal, } ); if (!response.ok) { throw new Error(`Token fetch failed: ${response.status}`); } const { data } = await response.json(); return data.token; } private async establishConnection(): Promise { return new Promise((resolve, reject) => { const params = new URLSearchParams({ sample_rate: this.config.sampleRate.toString(), account_id: this.config.accountId, api_token: this.token!, }); if (this.config.language) { params.set('language', this.config.language); } const url = `wss://api.sully.ai/v1/audio/transcriptions/stream?${params}`; this.ws = new WebSocket(url); const connectionTimeout = setTimeout(() => { this.ws?.close(); reject(new Error('Connection timeout')); }, 10000); this.ws.onopen = () => { clearTimeout(connectionTimeout); this.state = 'connected'; this.reconnectAttempt = 0; this.emit('stateChange', this.state); this.emit('connected'); // Flush buffered audio this.flushAudioBuffer(); resolve(); }; this.ws.onmessage = (event) => { this.handleMessage(event.data); }; this.ws.onerror = (error) => { clearTimeout(connectionTimeout); this.emit('error', error); }; this.ws.onclose = (event) => { clearTimeout(connectionTimeout); this.handleDisconnect(event); if (this.state === 'connecting') { reject(new Error(`Connection closed: ${event.code}`)); } }; }); } private handleMessage(data: string): void { try { const message = JSON.parse(data); if (message.error) { this.emit('error', new Error(message.error)); return; } if (message.text !== undefined) { this.segments[this.currentSegmentIndex] = message.text; const segment: TranscriptionSegment = { index: this.currentSegmentIndex, text: message.text, isFinal: message.isFinal ?? false, }; this.emit('transcription', segment); if (message.isFinal) { this.currentSegmentIndex++; } } } catch (error) { this.emit('error', new Error(`Failed to parse message: ${data}`)); } } private async handleDisconnect(event: CloseEvent): Promise { const wasConnected = this.state === 'connected'; this.ws = null; // Normal closure or intentional disconnect if (event.code === 1000 || this.state === 'disconnected') { this.state = 'disconnected'; this.emit('stateChange', this.state); this.emit('disconnected', { code: event.code, reason: event.reason }); return; } // Unexpected disconnect - attempt reconnection if (wasConnected && this.reconnectAttempt < this.config.maxReconnectAttempts) { await this.attemptReconnect(); } else { this.state = 'disconnected'; this.emit('stateChange', this.state); this.emit('disconnected', { code: event.code, reason: event.reason, reconnectFailed: true, }); } } private async attemptReconnect(): Promise { this.state = 'reconnecting'; this.emit('stateChange', this.state); const delay = this.calculateBackoff(); this.emit('reconnecting', { attempt: this.reconnectAttempt + 1, maxAttempts: this.config.maxReconnectAttempts, delayMs: delay, }); await this.sleep(delay); this.reconnectAttempt++; try { // Get fresh token for reconnection this.token = await this.fetchToken(); await this.establishConnection(); } catch (error) { this.emit('error', error); // Will trigger another reconnect attempt via onclose handler } } private calculateBackoff(): number { const exponentialDelay = this.config.baseReconnectDelayMs * Math.pow(2, this.reconnectAttempt); const cappedDelay = Math.min( exponentialDelay, this.config.maxReconnectDelayMs ); const jitter = cappedDelay * Math.random() * 0.25; return Math.floor(cappedDelay + jitter); } sendAudio(audioData: ArrayBuffer | Uint8Array): void { const base64Audio = this.arrayBufferToBase64(audioData); if (this.state === 'connected' && this.ws?.readyState === WebSocket.OPEN) { // Send immediately if connected this.ws.send(JSON.stringify({ audio: base64Audio })); } else if ( this.state === 'reconnecting' || this.state === 'connecting' ) { // Buffer audio during reconnection this.bufferAudio(base64Audio); } // Drop audio if disconnected (not reconnecting) } private bufferAudio(base64Audio: string): void { this.audioBuffer.push(base64Audio); // Prevent unbounded buffer growth while (this.audioBuffer.length > this.config.audioBufferMaxSize) { this.audioBuffer.shift(); this.emit('bufferOverflow'); } } private flushAudioBuffer(): void { if (this.audioBuffer.length === 0) return; const bufferedCount = this.audioBuffer.length; this.emit('bufferFlush', { count: bufferedCount }); for (const base64Audio of this.audioBuffer) { if (this.ws?.readyState === WebSocket.OPEN) { this.ws.send(JSON.stringify({ audio: base64Audio })); } } this.audioBuffer = []; } private arrayBufferToBase64(buffer: ArrayBuffer | Uint8Array): string { const bytes = buffer instanceof Uint8Array ? buffer : new Uint8Array(buffer); let binary = ''; for (let i = 0; i < bytes.byteLength; i++) { binary += String.fromCharCode(bytes[i]); } return btoa(binary); } private sleep(ms: number): Promise { return new Promise((resolve) => setTimeout(resolve, ms)); } getTranscript(): string { return this.segments.join(' '); } getState(): ConnectionState { return this.state; } disconnect(): void { this.state = 'disconnected'; this.abortController?.abort(); this.ws?.close(1000, 'Client disconnect'); this.ws = null; this.audioBuffer = []; } } // Usage example const stream = new ProductionTranscriptionStream({ accountId: process.env.SULLY_ACCOUNT_ID!, apiKey: process.env.SULLY_API_KEY!, sampleRate: 16000, language: 'en', }); stream.on('stateChange', (state) => { console.log(`Connection state: ${state}`); }); stream.on('transcription', (segment) => { if (segment.isFinal) { console.log(`[Final] ${segment.text}`); } else { console.log(`[Interim] ${segment.text}`); } }); stream.on('reconnecting', ({ attempt, maxAttempts, delayMs }) => { console.log(`Reconnecting (${attempt}/${maxAttempts}) in ${delayMs}ms`); }); stream.on('error', (error) => { console.error('Stream error:', error); }); await stream.connect(); // Send audio from microphone, file, etc. // stream.sendAudio(audioChunk); // When done // stream.disconnect(); ``` ```python Python theme={null} import asyncio import websockets import json import base64 import random import logging from dataclasses import dataclass, field from enum import Enum from typing import Optional, Callable, List import aiohttp logger = logging.getLogger(__name__) class ConnectionState(Enum): DISCONNECTED = "disconnected" CONNECTING = "connecting" CONNECTED = "connected" RECONNECTING = "reconnecting" @dataclass class StreamConfig: account_id: str api_key: str sample_rate: int language: str = "en" max_reconnect_attempts: int = 5 base_reconnect_delay_ms: int = 1000 max_reconnect_delay_ms: int = 30000 audio_buffer_max_size: int = 100 @dataclass class TranscriptionSegment: index: int text: str is_final: bool class ProductionTranscriptionStream: def __init__(self, config: StreamConfig): self.config = config self._ws: Optional[websockets.WebSocketClientProtocol] = None self._state = ConnectionState.DISCONNECTED self._reconnect_attempt = 0 self._audio_buffer: List[str] = [] self._segments: List[str] = [] self._current_segment_index = 0 self._token: Optional[str] = None self._shutdown = False # Event callbacks self.on_state_change: Optional[Callable[[ConnectionState], None]] = None self.on_transcription: Optional[Callable[[TranscriptionSegment], None]] = None self.on_error: Optional[Callable[[Exception], None]] = None self.on_reconnecting: Optional[Callable[[int, int, int], None]] = None self.on_connected: Optional[Callable[[], None]] = None self.on_disconnected: Optional[Callable[[int, str, bool], None]] = None @property def state(self) -> ConnectionState: return self._state @state.setter def state(self, value: ConnectionState): self._state = value if self.on_state_change: self.on_state_change(value) async def connect(self) -> None: """Establish connection to the transcription stream.""" self._shutdown = False self.state = ConnectionState.CONNECTING try: self._token = await self._fetch_token() await self._establish_connection() except Exception as e: self.state = ConnectionState.DISCONNECTED raise e async def _fetch_token(self) -> str: """Fetch a streaming token from the API.""" async with aiohttp.ClientSession() as session: async with session.post( "https://api.sully.ai/v1/audio/transcriptions/stream/token", headers={ "X-API-Key": self.config.api_key, "X-Account-Id": self.config.account_id, }, ) as response: if not response.ok: raise Exception(f"Token fetch failed: {response.status}") data = await response.json() return data["data"]["token"] async def _establish_connection(self) -> None: """Establish WebSocket connection.""" params = { "sample_rate": str(self.config.sample_rate), "account_id": self.config.account_id, "api_token": self._token, } if self.config.language: params["language"] = self.config.language query = "&".join(f"{k}={v}" for k, v in params.items()) url = f"wss://api.sully.ai/v1/audio/transcriptions/stream?{query}" try: self._ws = await asyncio.wait_for( websockets.connect(url), timeout=10.0 ) self.state = ConnectionState.CONNECTED self._reconnect_attempt = 0 if self.on_connected: self.on_connected() # Flush buffered audio await self._flush_audio_buffer() # Start receiving messages asyncio.create_task(self._receive_loop()) except asyncio.TimeoutError: raise Exception("Connection timeout") async def _receive_loop(self) -> None: """Continuously receive messages from WebSocket.""" try: async for message in self._ws: self._handle_message(message) except websockets.ConnectionClosed as e: await self._handle_disconnect(e.code, e.reason or "") except Exception as e: if self.on_error: self.on_error(e) await self._handle_disconnect(1006, str(e)) def _handle_message(self, data: str) -> None: """Process incoming WebSocket message.""" try: message = json.loads(data) if "error" in message: if self.on_error: self.on_error(Exception(message["error"])) return if "text" in message: # Extend segments list if needed while len(self._segments) <= self._current_segment_index: self._segments.append("") self._segments[self._current_segment_index] = message["text"] segment = TranscriptionSegment( index=self._current_segment_index, text=message["text"], is_final=message.get("isFinal", False), ) if self.on_transcription: self.on_transcription(segment) if message.get("isFinal"): self._current_segment_index += 1 except json.JSONDecodeError as e: if self.on_error: self.on_error(Exception(f"Failed to parse message: {data}")) async def _handle_disconnect(self, code: int, reason: str) -> None: """Handle WebSocket disconnection.""" was_connected = self.state == ConnectionState.CONNECTED self._ws = None # Normal closure or intentional disconnect if code == 1000 or self._shutdown: self.state = ConnectionState.DISCONNECTED if self.on_disconnected: self.on_disconnected(code, reason, False) return # Unexpected disconnect - attempt reconnection if was_connected and self._reconnect_attempt < self.config.max_reconnect_attempts: await self._attempt_reconnect() else: self.state = ConnectionState.DISCONNECTED if self.on_disconnected: self.on_disconnected(code, reason, True) async def _attempt_reconnect(self) -> None: """Attempt to reconnect with exponential backoff.""" self.state = ConnectionState.RECONNECTING delay_ms = self._calculate_backoff() if self.on_reconnecting: self.on_reconnecting( self._reconnect_attempt + 1, self.config.max_reconnect_attempts, delay_ms, ) await asyncio.sleep(delay_ms / 1000) self._reconnect_attempt += 1 try: self._token = await self._fetch_token() await self._establish_connection() except Exception as e: if self.on_error: self.on_error(e) # Will trigger another reconnect attempt via disconnect handler def _calculate_backoff(self) -> int: """Calculate reconnection delay with exponential backoff and jitter.""" exponential_delay = self.config.base_reconnect_delay_ms * ( 2 ** self._reconnect_attempt ) capped_delay = min(exponential_delay, self.config.max_reconnect_delay_ms) jitter = capped_delay * random.random() * 0.25 return int(capped_delay + jitter) async def send_audio(self, audio_data: bytes) -> None: """Send audio data to the stream.""" base64_audio = base64.b64encode(audio_data).decode("utf-8") if self.state == ConnectionState.CONNECTED and self._ws: await self._ws.send(json.dumps({"audio": base64_audio})) elif self.state in (ConnectionState.RECONNECTING, ConnectionState.CONNECTING): self._buffer_audio(base64_audio) # Drop audio if disconnected (not reconnecting) def _buffer_audio(self, base64_audio: str) -> None: """Buffer audio during reconnection.""" self._audio_buffer.append(base64_audio) # Prevent unbounded buffer growth while len(self._audio_buffer) > self.config.audio_buffer_max_size: self._audio_buffer.pop(0) logger.warning("Audio buffer overflow - dropping oldest chunk") async def _flush_audio_buffer(self) -> None: """Send buffered audio after reconnection.""" if not self._audio_buffer: return buffered_count = len(self._audio_buffer) logger.info(f"Flushing {buffered_count} buffered audio chunks") for base64_audio in self._audio_buffer: if self._ws: await self._ws.send(json.dumps({"audio": base64_audio})) self._audio_buffer = [] def get_transcript(self) -> str: """Get the full transcript assembled from all segments.""" return " ".join(self._segments) async def disconnect(self) -> None: """Gracefully disconnect from the stream.""" self._shutdown = True self.state = ConnectionState.DISCONNECTED if self._ws: await self._ws.close(1000, "Client disconnect") self._ws = None self._audio_buffer = [] # Usage example async def main(): stream = ProductionTranscriptionStream( StreamConfig( account_id=os.environ["SULLY_ACCOUNT_ID"], api_key=os.environ["SULLY_API_KEY"], sample_rate=16000, language="en", ) ) def on_state_change(state: ConnectionState): print(f"Connection state: {state.value}") def on_transcription(segment: TranscriptionSegment): prefix = "[Final]" if segment.is_final else "[Interim]" print(f"{prefix} {segment.text}") def on_reconnecting(attempt: int, max_attempts: int, delay_ms: int): print(f"Reconnecting ({attempt}/{max_attempts}) in {delay_ms}ms") def on_error(error: Exception): print(f"Stream error: {error}") stream.on_state_change = on_state_change stream.on_transcription = on_transcription stream.on_reconnecting = on_reconnecting stream.on_error = on_error await stream.connect() # Send audio from microphone, file, etc. # await stream.send_audio(audio_chunk) # When done # await stream.disconnect() if __name__ == "__main__": import os asyncio.run(main()) ``` ### Error Recovery Strategies | Error | Recovery Strategy | | ----------------------- | ---------------------------------------------------------------------------------------- | | Connection timeout | Retry with backoff, check network | | Token expired (401) | Fetch new token, reconnect | | Rate limited (429) | Use `Retry-After` header, increase backoff | | Server error (5xx) | Retry with backoff | | WebSocket error message | Surface to caller, keep listening for follow-up messages, reconnect if the socket closes | | Invalid audio format | Check sample rate, encoding | | Network disconnect | Reconnect with buffered audio | Always implement a maximum reconnection limit. Infinite reconnection loops can drain device batteries and create unnecessary server load. ## Language Support Sully.ai supports transcription in multiple languages using [BCP47 language tags](https://en.wikipedia.org/wiki/IETF_language_tag). See [Supported Languages](/api-reference/audio-transcriptions/languages) for the full list of **89** accepted locale codes. ### Common language tags | Language | Tags | | ---------- | ---------------------------------------------------------------------- | | English | `en`, `en-US`, `en-CA`, `en-IE`, `en-AU`, `en-GB`, `en-NZ`, `en-IN` | | Spanish | `es`, `es-419` | | Chinese | `zh`, `zh-CN`, `zh-Hans`, `zh-TW`, `zh-Hant`, `zh-HK` | | French | `fr`, `fr-CA` | | German | `de`, `de-CH` | | Portuguese | `pt`, `pt-BR`, `pt-PT` | | Japanese | `ja` | | Korean | `ko`, `ko-KR` | | Arabic | `ar`, `ar-EG`, `ar-SA`, and other regional `ar-*` tags (see full list) | ### Multilingual Mode For conversations that switch between languages, use `language=multi`: ```typescript TypeScript SDK theme={null} // File upload with multilingual support const transcription = await client.audio.transcriptions.create({ audio: fs.createReadStream('multilingual-visit.mp3'), language: 'multi', }); ``` ```python Python SDK theme={null} # File upload with multilingual support transcription = client.audio.transcriptions.create( audio=open("multilingual-visit.mp3", "rb"), language="multi" ) ``` ```bash HTTP (Upload) theme={null} curl -X POST "https://api.sully.ai/v2/audio/transcriptions?language=multi" \ -H "X-API-Key: ${SULLY_API_KEY}" \ -H "X-Account-Id: ${SULLY_ACCOUNT_ID}" \ -F "audio=@./multilingual-visit.mp3" ``` ### Language in Streaming Specify language when connecting to the WebSocket: ``` wss://api.sully.ai/v1/audio/transcriptions/stream?sample_rate=16000&account_id={id}&api_token={token}&language=es ``` For most locale codes, audio in other languages is filtered out. Base tags such as `es`, `fr`, and `de` route to multilingual code-switching instead — see [automatic multilingual routing](/api-reference/audio-transcriptions/languages#automatic-multilingual-routing). Use `multi` on streaming when speakers switch languages freely. ## Choosing Upload vs Stream Use this decision matrix to select the right approach: | Criterion | File Upload | Real-time Stream | | ----------------------------- | ------------------------------------ | ---------------------------------- | | **Use Case** | Pre-recorded audio, batch processing | Live patient visits | | **Latency Requirement** | Seconds to minutes acceptable | Immediate feedback needed | | **File Size** | Any size up to 100MB | N/A (continuous stream) | | **Network Reliability** | Single request | Requires stable connection | | **Implementation Complexity** | Simple (HTTP upload + polling) | Complex (WebSocket + reconnection) | | **Offline Support** | Upload when online | Requires active connection | ### When to Use File Upload * Processing recorded audio from devices or archives * Batch transcription of multiple files * Integration with systems that produce audio files * Environments with unreliable network connectivity (upload when stable) * Backend processing pipelines ### When to Use Real-time Streaming * Live transcription during patient visits * Providing immediate visual feedback to clinicians * Interactive applications where users see text as they speak * Reducing perceived latency in clinical workflows * Mobile applications with microphone access Many applications use both approaches: real-time streaming for live visits with immediate feedback, and file upload for processing any recordings that were captured offline. ## Next Steps Convert transcriptions into structured clinical notes Get notified when transcriptions complete Full SDK reference for Node.js applications Full SDK reference for Python applications