Skip to main content

Overview

Sully.ai provides two approaches for converting patient conversations to text: Both approaches produce the same high-quality medical transcription output that can be passed to note generation.

File Upload

Upload pre-recorded audio files for asynchronous transcription. This approach is ideal for batch processing, large files, or when real-time feedback is not required.

Supported Formats

Maximum file size: 100MB. For larger files, consider splitting into segments or using real-time streaming.

Upload and Poll

File transcription is asynchronous. Submit your file, then poll for completion.

Dictation Formatting

If you want prerecorded transcript output formatted for dictation workflows, add the optional dictation field and set it to true. If omitted, dictation defaults to false.

Status Lifecycle

For production applications, use webhooks instead of polling to receive notifications when transcription completes.

Real-time Streaming

Stream audio in real-time during patient visits for immediate transcription feedback. This approach uses WebSockets to send audio chunks and receive transcription segments as they are processed.

Connection Flow

Get a Streaming Token

Before connecting to the WebSocket, obtain a short-lived token:

WebSocket URL

Connect to the streaming endpoint with your token and audio parameters:
For raw, headerless audio, send both encoding and sample_rate. For containerized audio, omit encoding. If sample_rate is omitted, the current streaming service still defaults it to 16000.

Message Format

Stream ready:
Sending audio:
Receiving transcription:
Error message:
  • Wait for the status: connected message before sending audio
  • text: The transcribed text for the current segment
  • is_final: Canonical finality flag for the current segment
  • isFinal: Compatibility alias for is_final
  • type: "error" indicates a transcription problem. Some runtime errors are non-terminal, while other failures are followed by socket closure.

Basic WebSocket Connection

Production Streaming

Real-time audio streaming in production requires handling network interruptions, reconnection, and audio buffering. This section provides battle-tested patterns for reliable streaming.

Key Challenges

  1. Network interruptions - Mobile networks and WiFi can drop unexpectedly
  2. Token expiration - Streaming tokens have limited validity
  3. Audio continuity - Buffering audio during reconnection to prevent data loss
  4. State recovery - Resuming transcription context after reconnection
  5. Error frames - Some server error messages are non-terminal, while others precede disconnects

Reconnection with Exponential Backoff

Never reconnect immediately after a failure. Use exponential backoff with jitter to prevent thundering herd problems:

Production WebSocket Implementation

The following implementation handles reconnection, audio buffering, and error recovery:
This example reconnects on close events and connection failures. In your own implementation, surface type: "error" messages immediately, but do not assume every error frame is terminal. Some runtime errors are followed by later stream messages, while other failures are followed by socket closure.

Error Recovery Strategies

Always implement a maximum reconnection limit. Infinite reconnection loops can drain device batteries and create unnecessary server load.

Language Support

Sully.ai supports transcription in multiple languages using BCP47 language tags. See Supported Languages for the full list of 89 accepted locale codes.

Common language tags

Multilingual Mode

For conversations that switch between languages, use language=multi:

Language in Streaming

Specify language when connecting to the WebSocket:
For most locale codes, audio in other languages is filtered out. Base tags such as es, fr, and de route to multilingual code-switching instead — see automatic multilingual routing. Use multi on streaming when speakers switch languages freely.

Choosing Upload vs Stream

Use this decision matrix to select the right approach:

When to Use File Upload

  • Processing recorded audio from devices or archives
  • Batch transcription of multiple files
  • Integration with systems that produce audio files
  • Environments with unreliable network connectivity (upload when stable)
  • Backend processing pipelines

When to Use Real-time Streaming

  • Live transcription during patient visits
  • Providing immediate visual feedback to clinicians
  • Interactive applications where users see text as they speak
  • Reducing perceived latency in clinical workflows
  • Mobile applications with microphone access
Many applications use both approaches: real-time streaming for live visits with immediate feedback, and file upload for processing any recordings that were captured offline.

Next Steps

Generate Notes

Convert transcriptions into structured clinical notes

Webhooks

Get notified when transcriptions complete

TypeScript SDK

Full SDK reference for Node.js applications

Python SDK

Full SDK reference for Python applications