Overview
Sully.ai provides two approaches for converting patient conversations to text:| Approach | Best For | Latency |
|---|---|---|
| File Upload | Pre-recorded audio, batch processing, large files | Async (seconds to minutes) |
| Real-time Streaming | Live visits, immediate feedback, interactive transcription | Real-time (~200ms) |
File Upload
Upload pre-recorded audio files for asynchronous transcription. This approach is ideal for batch processing, large files, or when real-time feedback is not required.Supported Formats
| Format | MIME Type | Extension |
|---|---|---|
| WAV | audio/wav | .wav |
| MP3 | audio/mpeg | .mp3 |
| FLAC | audio/flac | .flac |
| OGG | audio/ogg | .ogg |
| WebM | audio/webm | .webm |
| MP4 | audio/mp4 | .mp4 |
| M4A | audio/mp4 | .m4a |
| AAC | audio/aac | .aac |
| Opus | audio/opus | .opus |
Maximum file size: 100MB. For larger files, consider splitting into segments or using real-time streaming.
Upload and Poll
File transcription is asynchronous. Submit your file, then poll for completion.import SullyAI from '@sullyai/sullyai';
import * as fs from 'fs';
const client = new SullyAI();
// 1. Upload audio file
const transcription = await client.audio.transcriptions.create({
audio: fs.createReadStream('patient-visit.mp3'),
});
console.log(`Transcription ID: ${transcription.transcriptionId}`);
// 2. Poll until complete
let result = await client.audio.transcriptions.retrieve(
transcription.transcriptionId
);
while (result.status === 'STATUS_PROCESSING') {
await new Promise((resolve) => setTimeout(resolve, 2000));
result = await client.audio.transcriptions.retrieve(
transcription.transcriptionId
);
}
if (result.status === 'STATUS_ERROR') {
throw new Error('Transcription failed');
}
console.log('Transcript:', result.payload?.transcription);
import time
from sullyai import SullyAI
client = SullyAI()
# 1. Upload audio file
with open("patient-visit.mp3", "rb") as audio_file:
transcription = client.audio.transcriptions.create(audio=audio_file)
print(f"Transcription ID: {transcription.transcription_id}")
# 2. Poll until complete
while True:
result = client.audio.transcriptions.retrieve(transcription.transcription_id)
if result.status == "STATUS_SUCCEEDED":
print(f"Transcript: {result.payload.transcription}")
break
elif result.status == "STATUS_ERROR":
raise Exception("Transcription failed")
time.sleep(2)
# 1. Upload audio file
curl -X POST "https://api.sully.ai/v2/audio/transcriptions" \
-H "X-API-Key: ${SULLY_API_KEY}" \
-H "X-Account-Id: ${SULLY_ACCOUNT_ID}" \
-F "audio=@./patient-visit.mp3"
# Response: { "data": { "transcriptionId": "tr_abc123" } }
# 2. Poll for completion
curl -X GET "https://api.sully.ai/v2/audio/transcriptions/tr_abc123" \
-H "X-API-Key: ${SULLY_API_KEY}" \
-H "X-Account-Id: ${SULLY_ACCOUNT_ID}"
# When complete:
# { "data": { "status": "completed", "payload": { "transcription": "..." } } }
Dictation Formatting
If you want prerecorded transcript output formatted for dictation workflows, add the optionaldictation field and set it to true. If omitted,
dictation defaults to false.
import { readFile } from 'fs/promises';
const audioBytes = await readFile('./patient-visit.mp3');
const formData = new FormData();
formData.append(
'audio',
new File([audioBytes], 'patient-visit.mp3', { type: 'audio/mpeg' })
);
formData.append('dictation', 'true');
await fetch('https://api.sully.ai/v2/audio/transcriptions', {
method: 'POST',
headers: {
'X-API-Key': process.env.SULLY_API_KEY!,
'X-Account-Id': process.env.SULLY_ACCOUNT_ID!,
},
body: formData,
});
import os
import requests
with open("patient-visit.mp3", "rb") as audio_file:
response = requests.post(
"https://api.sully.ai/v2/audio/transcriptions",
headers={
"X-API-Key": os.environ["SULLY_API_KEY"],
"X-Account-Id": os.environ["SULLY_ACCOUNT_ID"],
},
files={
"audio": ("patient-visit.mp3", audio_file, "audio/mpeg"),
},
data={
"dictation": "true",
},
)
response.raise_for_status()
curl -X POST "https://api.sully.ai/v2/audio/transcriptions" \
-H "X-API-Key: ${SULLY_API_KEY}" \
-H "X-Account-Id: ${SULLY_ACCOUNT_ID}" \
-F "audio=@./patient-visit.mp3" \
-F "dictation=true"
Status Lifecycle
| Status | Description |
|---|---|
pending | Request received, queued for processing |
processing | Actively being transcribed |
completed | Transcription ready in payload.transcription |
failed | An error occurred |
For production applications, use webhooks instead of polling to receive notifications when transcription completes.
Real-time Streaming
Stream audio in real-time during patient visits for immediate transcription feedback. This approach uses WebSockets to send audio chunks and receive transcription segments as they are processed.Connection Flow
1. Get token POST /v1/audio/transcriptions/stream/token
2. Connect wss://api.sully.ai/v1/audio/transcriptions/stream?...
3. Wait for { "type": "status", "status": "connected" }
4. Send audio { "audio": "<base64-encoded-audio>" }
5. Receive status, transcript, and error messages
6. Close ws.close()
Get a Streaming Token
Before connecting to the WebSocket, obtain a short-lived token:const tokenResponse = await fetch(
'https://api.sully.ai/v1/audio/transcriptions/stream/token',
{
method: 'POST',
headers: {
'X-API-Key': process.env.SULLY_API_KEY!,
'X-Account-Id': process.env.SULLY_ACCOUNT_ID!,
},
}
);
const { data: { token } } = await tokenResponse.json();
import requests
import os
response = requests.post(
"https://api.sully.ai/v1/audio/transcriptions/stream/token",
headers={
"X-API-Key": os.environ["SULLY_API_KEY"],
"X-Account-Id": os.environ["SULLY_ACCOUNT_ID"],
}
)
token = response.json()["data"]["token"]
curl -X POST "https://api.sully.ai/v1/audio/transcriptions/stream/token" \
-H "X-API-Key: ${SULLY_API_KEY}" \
-H "X-Account-Id: ${SULLY_ACCOUNT_ID}"
# Response: { "data": { "token": "eyJ..." } }
WebSocket URL
Connect to the streaming endpoint with your token and audio parameters:wss://api.sully.ai/v1/audio/transcriptions/stream?sample_rate=16000&account_id={accountId}&api_token={token}
| Parameter | Required | Description |
|---|---|---|
sample_rate | No | Audio sample rate in Hz (e.g., 16000, 44100). If omitted, the streaming service currently defaults to 16000. For raw, headerless audio, send the actual sample rate explicitly. |
account_id | Yes | Your Sully account ID |
api_token | Yes | Token from the stream token endpoint |
language | No | BCP47 language tag (e.g., en, es, multi) |
dictation | No | Set to true to request dictation-oriented transcript formatting |
For raw, headerless audio, send both
encoding and sample_rate. For
containerized audio, omit encoding. If sample_rate is omitted, the current
streaming service still defaults it to 16000.Message Format
Stream ready:{
"type": "status",
"status": "connected",
"timestamp": "2026-04-22T14:32:07.123Z"
}
{ "audio": "<base64-encoded-audio-chunk>" }
{
"type": "transcript",
"text": "The patient reports feeling tired",
"isFinal": false,
"is_final": false
}
{
"type": "error",
"error": "An error occurred during transcription",
"timestamp": "2026-04-22T14:32:12.123Z"
}
- Wait for the
status: connectedmessage before sending audio text: The transcribed text for the current segmentis_final: Canonical finality flag for the current segmentisFinal: Compatibility alias foris_finaltype: "error"indicates a transcription problem. Some runtime errors are non-terminal, while other failures are followed by socket closure.
Basic WebSocket Connection
// Get streaming token first (see above)
const token = await getStreamingToken();
const accountId = process.env.SULLY_ACCOUNT_ID!;
// Connect to WebSocket
const ws = new WebSocket(
`wss://api.sully.ai/v1/audio/transcriptions/stream?sample_rate=16000&account_id=${accountId}&api_token=${token}`
);
// Track transcription segments
const segments: string[] = [];
let currentIndex = 0;
let streamReady = false;
ws.onopen = () => {
console.log('Socket opened, waiting for stream readiness...');
};
ws.onmessage = (event) => {
const data = JSON.parse(event.data);
if (data.type === 'status') {
streamReady = data.status === 'connected';
console.log(`Stream status: ${data.status}`);
return;
}
if (data.type === 'error') {
console.warn('Transcription stream error:', data.error);
return;
}
if (data.type === 'transcript' && data.text) {
segments[currentIndex] = data.text;
const isFinal = data.is_final ?? data.isFinal ?? false;
if (isFinal) {
console.log(`Segment ${currentIndex}: ${data.text}`);
currentIndex++;
}
}
};
ws.onerror = (error) => {
console.error('WebSocket error:', error);
};
ws.onclose = (event) => {
console.log(`Connection closed: ${event.code} ${event.reason}`);
const fullTranscript = segments.join(' ');
console.log('Full transcript:', fullTranscript);
};
// Send audio data (from microphone, file, etc.)
function sendAudio(audioBuffer: ArrayBuffer) {
if (!streamReady) {
console.warn('Stream not ready yet; wait for the connected status message.');
return;
}
const base64Audio = btoa(
String.fromCharCode(...new Uint8Array(audioBuffer))
);
ws.send(JSON.stringify({ audio: base64Audio }));
}
import asyncio
import websockets
import json
import base64
import os
async def stream_transcription():
# Get streaming token first (see above)
token = await get_streaming_token()
account_id = os.environ["SULLY_ACCOUNT_ID"]
url = f"wss://api.sully.ai/v1/audio/transcriptions/stream?sample_rate=16000&account_id={account_id}&api_token={token}"
segments = []
current_index = 0
stream_ready = False
async with websockets.connect(url) as ws:
print("Socket opened, waiting for stream readiness...")
async def receive_messages():
nonlocal current_index, stream_ready
async for message in ws:
data = json.loads(message)
if data.get("type") == "status":
stream_ready = data.get("status") == "connected"
print(f"Stream status: {data.get('status')}")
continue
if data.get("type") == "error":
print(f"Transcription stream error: {data.get('error')}")
continue
if data.get("type") == "transcript" and "text" in data:
# Extend segments list if needed
while len(segments) <= current_index:
segments.append("")
segments[current_index] = data["text"]
is_final = data.get("is_final", data.get("isFinal", False))
if is_final:
print(f"Segment {current_index}: {data['text']}")
current_index += 1
async def send_audio(audio_data: bytes):
if not stream_ready:
print("Stream not ready yet; wait for the connected status message.")
return
base64_audio = base64.b64encode(audio_data).decode("utf-8")
await ws.send(json.dumps({"audio": base64_audio}))
# Start receiving messages
receive_task = asyncio.create_task(receive_messages())
# Send your audio data here
# await send_audio(audio_chunk)
# Wait for completion
await receive_task
full_transcript = " ".join(segments)
print(f"Full transcript: {full_transcript}")
asyncio.run(stream_transcription())
Production Streaming
Real-time audio streaming in production requires handling network interruptions, reconnection, and audio buffering. This section provides battle-tested patterns for reliable streaming.Key Challenges
- Network interruptions - Mobile networks and WiFi can drop unexpectedly
- Token expiration - Streaming tokens have limited validity
- Audio continuity - Buffering audio during reconnection to prevent data loss
- State recovery - Resuming transcription context after reconnection
- Error frames - Some server error messages are non-terminal, while others precede disconnects
Reconnection with Exponential Backoff
Never reconnect immediately after a failure. Use exponential backoff with jitter to prevent thundering herd problems:interface BackoffConfig {
baseDelayMs: number;
maxDelayMs: number;
maxAttempts: number;
}
function calculateBackoff(
attempt: number,
config: BackoffConfig
): number {
const exponentialDelay = config.baseDelayMs * Math.pow(2, attempt);
const cappedDelay = Math.min(exponentialDelay, config.maxDelayMs);
// Add jitter: random value between 0-25% of delay
const jitter = cappedDelay * Math.random() * 0.25;
return cappedDelay + jitter;
}
Production WebSocket Implementation
The following implementation handles reconnection, audio buffering, and error recovery:This example reconnects on close events and connection failures. In your own
implementation, surface
type: "error" messages immediately, but do not assume
every error frame is terminal. Some runtime errors are followed by later stream
messages, while other failures are followed by socket closure.import { EventEmitter } from 'events';
interface StreamConfig {
accountId: string;
apiKey: string;
sampleRate: number;
language?: string;
maxReconnectAttempts?: number;
baseReconnectDelayMs?: number;
maxReconnectDelayMs?: number;
audioBufferMaxSize?: number;
}
interface TranscriptionSegment {
index: number;
text: string;
isFinal: boolean;
}
type ConnectionState =
| 'disconnected'
| 'connecting'
| 'connected'
| 'reconnecting';
class ProductionTranscriptionStream extends EventEmitter {
private ws: WebSocket | null = null;
private state: ConnectionState = 'disconnected';
private reconnectAttempt = 0;
private audioBuffer: string[] = [];
private segments: string[] = [];
private currentSegmentIndex = 0;
private token: string | null = null;
private abortController: AbortController | null = null;
private readonly config: Required<StreamConfig>;
constructor(config: StreamConfig) {
super();
this.config = {
maxReconnectAttempts: 5,
baseReconnectDelayMs: 1000,
maxReconnectDelayMs: 30000,
audioBufferMaxSize: 100, // Buffer up to 100 audio chunks
language: 'en',
...config,
};
}
async connect(signal?: AbortSignal): Promise<void> {
if (signal?.aborted) {
throw new Error('Connection aborted');
}
this.abortController = new AbortController();
this.state = 'connecting';
this.emit('stateChange', this.state);
try {
// Get fresh token
this.token = await this.fetchToken();
await this.establishConnection();
} catch (error) {
this.state = 'disconnected';
this.emit('stateChange', this.state);
throw error;
}
}
private async fetchToken(): Promise<string> {
const response = await fetch(
'https://api.sully.ai/v1/audio/transcriptions/stream/token',
{
method: 'POST',
headers: {
'X-API-Key': this.config.apiKey,
'X-Account-Id': this.config.accountId,
},
signal: this.abortController?.signal,
}
);
if (!response.ok) {
throw new Error(`Token fetch failed: ${response.status}`);
}
const { data } = await response.json();
return data.token;
}
private async establishConnection(): Promise<void> {
return new Promise((resolve, reject) => {
const params = new URLSearchParams({
sample_rate: this.config.sampleRate.toString(),
account_id: this.config.accountId,
api_token: this.token!,
});
if (this.config.language) {
params.set('language', this.config.language);
}
const url = `wss://api.sully.ai/v1/audio/transcriptions/stream?${params}`;
this.ws = new WebSocket(url);
const connectionTimeout = setTimeout(() => {
this.ws?.close();
reject(new Error('Connection timeout'));
}, 10000);
this.ws.onopen = () => {
clearTimeout(connectionTimeout);
this.state = 'connected';
this.reconnectAttempt = 0;
this.emit('stateChange', this.state);
this.emit('connected');
// Flush buffered audio
this.flushAudioBuffer();
resolve();
};
this.ws.onmessage = (event) => {
this.handleMessage(event.data);
};
this.ws.onerror = (error) => {
clearTimeout(connectionTimeout);
this.emit('error', error);
};
this.ws.onclose = (event) => {
clearTimeout(connectionTimeout);
this.handleDisconnect(event);
if (this.state === 'connecting') {
reject(new Error(`Connection closed: ${event.code}`));
}
};
});
}
private handleMessage(data: string): void {
try {
const message = JSON.parse(data);
if (message.error) {
this.emit('error', new Error(message.error));
return;
}
if (message.text !== undefined) {
this.segments[this.currentSegmentIndex] = message.text;
const segment: TranscriptionSegment = {
index: this.currentSegmentIndex,
text: message.text,
isFinal: message.isFinal ?? false,
};
this.emit('transcription', segment);
if (message.isFinal) {
this.currentSegmentIndex++;
}
}
} catch (error) {
this.emit('error', new Error(`Failed to parse message: ${data}`));
}
}
private async handleDisconnect(event: CloseEvent): Promise<void> {
const wasConnected = this.state === 'connected';
this.ws = null;
// Normal closure or intentional disconnect
if (event.code === 1000 || this.state === 'disconnected') {
this.state = 'disconnected';
this.emit('stateChange', this.state);
this.emit('disconnected', { code: event.code, reason: event.reason });
return;
}
// Unexpected disconnect - attempt reconnection
if (wasConnected && this.reconnectAttempt < this.config.maxReconnectAttempts) {
await this.attemptReconnect();
} else {
this.state = 'disconnected';
this.emit('stateChange', this.state);
this.emit('disconnected', {
code: event.code,
reason: event.reason,
reconnectFailed: true,
});
}
}
private async attemptReconnect(): Promise<void> {
this.state = 'reconnecting';
this.emit('stateChange', this.state);
const delay = this.calculateBackoff();
this.emit('reconnecting', {
attempt: this.reconnectAttempt + 1,
maxAttempts: this.config.maxReconnectAttempts,
delayMs: delay,
});
await this.sleep(delay);
this.reconnectAttempt++;
try {
// Get fresh token for reconnection
this.token = await this.fetchToken();
await this.establishConnection();
} catch (error) {
this.emit('error', error);
// Will trigger another reconnect attempt via onclose handler
}
}
private calculateBackoff(): number {
const exponentialDelay =
this.config.baseReconnectDelayMs * Math.pow(2, this.reconnectAttempt);
const cappedDelay = Math.min(
exponentialDelay,
this.config.maxReconnectDelayMs
);
const jitter = cappedDelay * Math.random() * 0.25;
return Math.floor(cappedDelay + jitter);
}
sendAudio(audioData: ArrayBuffer | Uint8Array): void {
const base64Audio = this.arrayBufferToBase64(audioData);
if (this.state === 'connected' && this.ws?.readyState === WebSocket.OPEN) {
// Send immediately if connected
this.ws.send(JSON.stringify({ audio: base64Audio }));
} else if (
this.state === 'reconnecting' ||
this.state === 'connecting'
) {
// Buffer audio during reconnection
this.bufferAudio(base64Audio);
}
// Drop audio if disconnected (not reconnecting)
}
private bufferAudio(base64Audio: string): void {
this.audioBuffer.push(base64Audio);
// Prevent unbounded buffer growth
while (this.audioBuffer.length > this.config.audioBufferMaxSize) {
this.audioBuffer.shift();
this.emit('bufferOverflow');
}
}
private flushAudioBuffer(): void {
if (this.audioBuffer.length === 0) return;
const bufferedCount = this.audioBuffer.length;
this.emit('bufferFlush', { count: bufferedCount });
for (const base64Audio of this.audioBuffer) {
if (this.ws?.readyState === WebSocket.OPEN) {
this.ws.send(JSON.stringify({ audio: base64Audio }));
}
}
this.audioBuffer = [];
}
private arrayBufferToBase64(buffer: ArrayBuffer | Uint8Array): string {
const bytes = buffer instanceof Uint8Array ? buffer : new Uint8Array(buffer);
let binary = '';
for (let i = 0; i < bytes.byteLength; i++) {
binary += String.fromCharCode(bytes[i]);
}
return btoa(binary);
}
private sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
getTranscript(): string {
return this.segments.join(' ');
}
getState(): ConnectionState {
return this.state;
}
disconnect(): void {
this.state = 'disconnected';
this.abortController?.abort();
this.ws?.close(1000, 'Client disconnect');
this.ws = null;
this.audioBuffer = [];
}
}
// Usage example
const stream = new ProductionTranscriptionStream({
accountId: process.env.SULLY_ACCOUNT_ID!,
apiKey: process.env.SULLY_API_KEY!,
sampleRate: 16000,
language: 'en',
});
stream.on('stateChange', (state) => {
console.log(`Connection state: ${state}`);
});
stream.on('transcription', (segment) => {
if (segment.isFinal) {
console.log(`[Final] ${segment.text}`);
} else {
console.log(`[Interim] ${segment.text}`);
}
});
stream.on('reconnecting', ({ attempt, maxAttempts, delayMs }) => {
console.log(`Reconnecting (${attempt}/${maxAttempts}) in ${delayMs}ms`);
});
stream.on('error', (error) => {
console.error('Stream error:', error);
});
await stream.connect();
// Send audio from microphone, file, etc.
// stream.sendAudio(audioChunk);
// When done
// stream.disconnect();
import asyncio
import websockets
import json
import base64
import random
import logging
from dataclasses import dataclass, field
from enum import Enum
from typing import Optional, Callable, List
import aiohttp
logger = logging.getLogger(__name__)
class ConnectionState(Enum):
DISCONNECTED = "disconnected"
CONNECTING = "connecting"
CONNECTED = "connected"
RECONNECTING = "reconnecting"
@dataclass
class StreamConfig:
account_id: str
api_key: str
sample_rate: int
language: str = "en"
max_reconnect_attempts: int = 5
base_reconnect_delay_ms: int = 1000
max_reconnect_delay_ms: int = 30000
audio_buffer_max_size: int = 100
@dataclass
class TranscriptionSegment:
index: int
text: str
is_final: bool
class ProductionTranscriptionStream:
def __init__(self, config: StreamConfig):
self.config = config
self._ws: Optional[websockets.WebSocketClientProtocol] = None
self._state = ConnectionState.DISCONNECTED
self._reconnect_attempt = 0
self._audio_buffer: List[str] = []
self._segments: List[str] = []
self._current_segment_index = 0
self._token: Optional[str] = None
self._shutdown = False
# Event callbacks
self.on_state_change: Optional[Callable[[ConnectionState], None]] = None
self.on_transcription: Optional[Callable[[TranscriptionSegment], None]] = None
self.on_error: Optional[Callable[[Exception], None]] = None
self.on_reconnecting: Optional[Callable[[int, int, int], None]] = None
self.on_connected: Optional[Callable[[], None]] = None
self.on_disconnected: Optional[Callable[[int, str, bool], None]] = None
@property
def state(self) -> ConnectionState:
return self._state
@state.setter
def state(self, value: ConnectionState):
self._state = value
if self.on_state_change:
self.on_state_change(value)
async def connect(self) -> None:
"""Establish connection to the transcription stream."""
self._shutdown = False
self.state = ConnectionState.CONNECTING
try:
self._token = await self._fetch_token()
await self._establish_connection()
except Exception as e:
self.state = ConnectionState.DISCONNECTED
raise e
async def _fetch_token(self) -> str:
"""Fetch a streaming token from the API."""
async with aiohttp.ClientSession() as session:
async with session.post(
"https://api.sully.ai/v1/audio/transcriptions/stream/token",
headers={
"X-API-Key": self.config.api_key,
"X-Account-Id": self.config.account_id,
},
) as response:
if not response.ok:
raise Exception(f"Token fetch failed: {response.status}")
data = await response.json()
return data["data"]["token"]
async def _establish_connection(self) -> None:
"""Establish WebSocket connection."""
params = {
"sample_rate": str(self.config.sample_rate),
"account_id": self.config.account_id,
"api_token": self._token,
}
if self.config.language:
params["language"] = self.config.language
query = "&".join(f"{k}={v}" for k, v in params.items())
url = f"wss://api.sully.ai/v1/audio/transcriptions/stream?{query}"
try:
self._ws = await asyncio.wait_for(
websockets.connect(url),
timeout=10.0
)
self.state = ConnectionState.CONNECTED
self._reconnect_attempt = 0
if self.on_connected:
self.on_connected()
# Flush buffered audio
await self._flush_audio_buffer()
# Start receiving messages
asyncio.create_task(self._receive_loop())
except asyncio.TimeoutError:
raise Exception("Connection timeout")
async def _receive_loop(self) -> None:
"""Continuously receive messages from WebSocket."""
try:
async for message in self._ws:
self._handle_message(message)
except websockets.ConnectionClosed as e:
await self._handle_disconnect(e.code, e.reason or "")
except Exception as e:
if self.on_error:
self.on_error(e)
await self._handle_disconnect(1006, str(e))
def _handle_message(self, data: str) -> None:
"""Process incoming WebSocket message."""
try:
message = json.loads(data)
if "error" in message:
if self.on_error:
self.on_error(Exception(message["error"]))
return
if "text" in message:
# Extend segments list if needed
while len(self._segments) <= self._current_segment_index:
self._segments.append("")
self._segments[self._current_segment_index] = message["text"]
segment = TranscriptionSegment(
index=self._current_segment_index,
text=message["text"],
is_final=message.get("isFinal", False),
)
if self.on_transcription:
self.on_transcription(segment)
if message.get("isFinal"):
self._current_segment_index += 1
except json.JSONDecodeError as e:
if self.on_error:
self.on_error(Exception(f"Failed to parse message: {data}"))
async def _handle_disconnect(self, code: int, reason: str) -> None:
"""Handle WebSocket disconnection."""
was_connected = self.state == ConnectionState.CONNECTED
self._ws = None
# Normal closure or intentional disconnect
if code == 1000 or self._shutdown:
self.state = ConnectionState.DISCONNECTED
if self.on_disconnected:
self.on_disconnected(code, reason, False)
return
# Unexpected disconnect - attempt reconnection
if was_connected and self._reconnect_attempt < self.config.max_reconnect_attempts:
await self._attempt_reconnect()
else:
self.state = ConnectionState.DISCONNECTED
if self.on_disconnected:
self.on_disconnected(code, reason, True)
async def _attempt_reconnect(self) -> None:
"""Attempt to reconnect with exponential backoff."""
self.state = ConnectionState.RECONNECTING
delay_ms = self._calculate_backoff()
if self.on_reconnecting:
self.on_reconnecting(
self._reconnect_attempt + 1,
self.config.max_reconnect_attempts,
delay_ms,
)
await asyncio.sleep(delay_ms / 1000)
self._reconnect_attempt += 1
try:
self._token = await self._fetch_token()
await self._establish_connection()
except Exception as e:
if self.on_error:
self.on_error(e)
# Will trigger another reconnect attempt via disconnect handler
def _calculate_backoff(self) -> int:
"""Calculate reconnection delay with exponential backoff and jitter."""
exponential_delay = self.config.base_reconnect_delay_ms * (
2 ** self._reconnect_attempt
)
capped_delay = min(exponential_delay, self.config.max_reconnect_delay_ms)
jitter = capped_delay * random.random() * 0.25
return int(capped_delay + jitter)
async def send_audio(self, audio_data: bytes) -> None:
"""Send audio data to the stream."""
base64_audio = base64.b64encode(audio_data).decode("utf-8")
if self.state == ConnectionState.CONNECTED and self._ws:
await self._ws.send(json.dumps({"audio": base64_audio}))
elif self.state in (ConnectionState.RECONNECTING, ConnectionState.CONNECTING):
self._buffer_audio(base64_audio)
# Drop audio if disconnected (not reconnecting)
def _buffer_audio(self, base64_audio: str) -> None:
"""Buffer audio during reconnection."""
self._audio_buffer.append(base64_audio)
# Prevent unbounded buffer growth
while len(self._audio_buffer) > self.config.audio_buffer_max_size:
self._audio_buffer.pop(0)
logger.warning("Audio buffer overflow - dropping oldest chunk")
async def _flush_audio_buffer(self) -> None:
"""Send buffered audio after reconnection."""
if not self._audio_buffer:
return
buffered_count = len(self._audio_buffer)
logger.info(f"Flushing {buffered_count} buffered audio chunks")
for base64_audio in self._audio_buffer:
if self._ws:
await self._ws.send(json.dumps({"audio": base64_audio}))
self._audio_buffer = []
def get_transcript(self) -> str:
"""Get the full transcript assembled from all segments."""
return " ".join(self._segments)
async def disconnect(self) -> None:
"""Gracefully disconnect from the stream."""
self._shutdown = True
self.state = ConnectionState.DISCONNECTED
if self._ws:
await self._ws.close(1000, "Client disconnect")
self._ws = None
self._audio_buffer = []
# Usage example
async def main():
stream = ProductionTranscriptionStream(
StreamConfig(
account_id=os.environ["SULLY_ACCOUNT_ID"],
api_key=os.environ["SULLY_API_KEY"],
sample_rate=16000,
language="en",
)
)
def on_state_change(state: ConnectionState):
print(f"Connection state: {state.value}")
def on_transcription(segment: TranscriptionSegment):
prefix = "[Final]" if segment.is_final else "[Interim]"
print(f"{prefix} {segment.text}")
def on_reconnecting(attempt: int, max_attempts: int, delay_ms: int):
print(f"Reconnecting ({attempt}/{max_attempts}) in {delay_ms}ms")
def on_error(error: Exception):
print(f"Stream error: {error}")
stream.on_state_change = on_state_change
stream.on_transcription = on_transcription
stream.on_reconnecting = on_reconnecting
stream.on_error = on_error
await stream.connect()
# Send audio from microphone, file, etc.
# await stream.send_audio(audio_chunk)
# When done
# await stream.disconnect()
if __name__ == "__main__":
import os
asyncio.run(main())
Error Recovery Strategies
| Error | Recovery Strategy |
|---|---|
| Connection timeout | Retry with backoff, check network |
| Token expired (401) | Fetch new token, reconnect |
| Rate limited (429) | Use Retry-After header, increase backoff |
| Server error (5xx) | Retry with backoff |
| WebSocket error message | Surface to caller, keep listening for follow-up messages, reconnect if the socket closes |
| Invalid audio format | Check sample rate, encoding |
| Network disconnect | Reconnect with buffered audio |
Always implement a maximum reconnection limit. Infinite reconnection loops can drain device batteries and create unnecessary server load.
Language Support
Sully.ai supports transcription in multiple languages using BCP47 language tags. See Supported Languages for the full list of 89 accepted locale codes.Common language tags
| Language | Tags |
|---|---|
| English | en, en-US, en-CA, en-IE, en-AU, en-GB, en-NZ, en-IN |
| Spanish | es, es-419 |
| Chinese | zh, zh-CN, zh-Hans, zh-TW, zh-Hant, zh-HK |
| French | fr, fr-CA |
| German | de, de-CH |
| Portuguese | pt, pt-BR, pt-PT |
| Japanese | ja |
| Korean | ko, ko-KR |
| Arabic | ar, ar-EG, ar-SA, and other regional ar-* tags (see full list) |
Multilingual Mode
For conversations that switch between languages, uselanguage=multi:
// File upload with multilingual support
const transcription = await client.audio.transcriptions.create({
audio: fs.createReadStream('multilingual-visit.mp3'),
language: 'multi',
});
# File upload with multilingual support
transcription = client.audio.transcriptions.create(
audio=open("multilingual-visit.mp3", "rb"),
language="multi"
)
curl -X POST "https://api.sully.ai/v2/audio/transcriptions?language=multi" \
-H "X-API-Key: ${SULLY_API_KEY}" \
-H "X-Account-Id: ${SULLY_ACCOUNT_ID}" \
-F "audio=@./multilingual-visit.mp3"
Language in Streaming
Specify language when connecting to the WebSocket:wss://api.sully.ai/v1/audio/transcriptions/stream?sample_rate=16000&account_id={id}&api_token={token}&language=es
For most locale codes, audio in other languages is filtered out. Base tags such as
es, fr, and de route to multilingual code-switching instead — see automatic multilingual routing. Use multi on streaming when speakers switch languages freely.Choosing Upload vs Stream
Use this decision matrix to select the right approach:| Criterion | File Upload | Real-time Stream |
|---|---|---|
| Use Case | Pre-recorded audio, batch processing | Live patient visits |
| Latency Requirement | Seconds to minutes acceptable | Immediate feedback needed |
| File Size | Any size up to 100MB | N/A (continuous stream) |
| Network Reliability | Single request | Requires stable connection |
| Implementation Complexity | Simple (HTTP upload + polling) | Complex (WebSocket + reconnection) |
| Offline Support | Upload when online | Requires active connection |
When to Use File Upload
- Processing recorded audio from devices or archives
- Batch transcription of multiple files
- Integration with systems that produce audio files
- Environments with unreliable network connectivity (upload when stable)
- Backend processing pipelines
When to Use Real-time Streaming
- Live transcription during patient visits
- Providing immediate visual feedback to clinicians
- Interactive applications where users see text as they speak
- Reducing perceived latency in clinical workflows
- Mobile applications with microphone access
Many applications use both approaches: real-time streaming for live visits with immediate feedback, and file upload for processing any recordings that were captured offline.
Next Steps
Generate Notes
Convert transcriptions into structured clinical notes
Webhooks
Get notified when transcriptions complete
TypeScript SDK
Full SDK reference for Node.js applications
Python SDK
Full SDK reference for Python applications