Media Abstraction Layer

Media Abstraction Layer

The Media Abstraction Layer is a minimal, platform-neutral WebRTC contract that decouples the realtime-core engine from browser-native and mobile (react-native-webrtc) media implementations. It defines the MediaAdapter interface, the mobile rnMediaAdapter() implementation, and the media URL resolution helper used to fetch captured media assets.

Purpose and Scope

This page documents the media abstraction boundary of the realtime-core subsystem:

• The MediaAdapter interface contract (createPeerConnection, getUserMedia) and the CallKind type.
• How the adapter is injected into the engine via EngineConfig.mediaAdapter.
• The React Native mobile adapter rnMediaAdapter(), including Android runtime permission handling and the audio-only fallback chain.
• The resolveMediaUrl() utility that maps relative media URLs to absolute API endpoints.

Topics intentionally left to sibling pages: the SFU signaling protocol, call/offer/answer state machines, and WebRTC DataChannel handling belong to the realtime-core engine page; backend media storage and orphan-file cleanup belong to the media storage/cleanup pages (see Related Links).

Overview

The realtime-core engine is written once and must run on multiple platforms: web browsers expose WebRTC through global RTCPeerConnection / mediaDevices, while mobile apps use react-native-webrtc, whose APIs are similar but not identical and which require explicit Android runtime permission handling. Rather than scattering Platform.OS checks and native imports through the engine, the core defines a two-method MediaAdapter interface that is injected per platform at configuration time.

The abstraction serves three purposes:

1. Platform isolation — the engine never imports react-native or browser WebRTC globals directly; it only depends on the standard TypeScript WebRTC types declared by the adapter contract.
2. Graceful degradation — the mobile implementation bakes in a deliberate fallback chain: if a full video capture fails (e.g., camera permission denied), the call degrades to audio-only rather than failing entirely; if even audio capture fails, an empty MediaStream is returned so the signaling flow can still proceed.
3. Configurable transport — peer connection configuration (ICE servers) is owned by the adapter, not the engine, so TURN/STUN changes are a per-platform concern.

Key concepts:

• MediaAdapter — the injected contract with exactly two methods: a peer-connection factory and a media-capture factory.
• CallKind — 'audio' | 'video'; the engine tells the adapter whether the local user’s camera is required.
• EngineConfig.mediaAdapter — the dependency-injection slot; when omitted, the engine falls back to the browser’s native WebRTC globals.
• rnMediaAdapter() — the mobile adapter factory backed by react-native-webrtc.

Architecture

Component roles:

• RealtimeEngine consumes the adapter through EngineConfig; it calls createPeerConnection(roomId) when establishing an SFU/peer session and getUserMedia(kind) when the local user joins a call. Media it receives is pushed to the engine’s MediaCallbacks.
• MediaAdapter is the contract boundary. It is deliberately tiny (two methods) so that any platform can implement it with minimal glue code.
• Browser adapter is the default path for web builds — the engine falls back to native browser globals when no adapter is injected (the TypeScript types are shared, so the engine compiles against the same signatures).
• rnMediaAdapter() is the mobile implementation. It wraps react-native-webrtc types, casts them to the standard RTCPeerConnection / MediaStream types the engine expects, and adds Android permission handling and capture fallbacks.
• ensureMediaPermissions() is a mobile-only pre-flight check; react-native-webrtc does not request Android 6+ runtime permissions itself, so the adapter must request RECORD_AUDIO (always) and CAMERA (for video calls) before calling getUserMedia.

The design intent of the split: the engine owns call state and signaling; the adapter owns device and transport specifics. This keeps platform quirks (permission dialogs, ICE server policy, TURN regressions) out of the shared call-state machine and lets each platform evolve independently.

The MediaAdapter Contract

The contract lives in realtime-core/src/types.ts (vendored into streaming/src/lib/realtime-core/types.ts and streaming-frontend/lib/realtime-core/types.ts so the engine ships with each app bundle):

export type CallKind = 'audio' | 'video';

/** WebRTC media + DataChannels, injected per platform */
export interface MediaAdapter {
  createPeerConnection(roomId: string): RTCPeerConnection;
  getUserMedia(kind: CallKind): Promise<MediaStream>;
}

Source: types.ts

The interface is intentionally minimal — two factories, no lifecycle management, no signaling. The design rationale:

• createPeerConnection(roomId) is a factory keyed by room so the adapter can (in principle) reuse or tag connections per room. In the current mobile implementation the roomId argument is accepted but not used, which keeps the engine-side call site stable while leaving room for future per-room ICE policies.
• getUserMedia(kind) is a capture factory returning a platform MediaStream. It is asynchronous because permission prompts and device enumeration are async on every platform.
• Return types are the standard DOM WebRTC types (RTCPeerConnection, MediaStream). This is deliberate: the engine and its tests compile against the well-known browser type definitions, and the mobile adapter casts its react-native-webrtc objects to those types at the boundary (see below).

Injection Point: EngineConfig

export interface EngineConfig {
  /** gRPC-Web base URL, e.g. '/grpc' (Next proxy) or 'http://host:50051' */
  baseUrl: string;
  /** Optional bearer token provider; when omitted, auth is injected by a proxy/cookie */
  tokenProvider?: () => string | Promise<string>;
  /** Stream reconnect delay after drop (ms). Default 3000 */
  reconnectDelayMs?: number;
  onError?: (error: Error) => void;
  /** Injected WebRTC layer (web browser / mobile react-native-webrtc) */
  mediaAdapter?: MediaAdapter;
  onLocalStream?: (stream: MediaStream) => void;
  /** Remote media stream plus the participant id it belongs to (stream.id from the SFU msid). */
  onRemoteStream?: (stream: MediaStream, participantId?: string, version?: number) => void;
}

Source: types.ts

mediaAdapter is optional: web builds omit it and let the engine use native browser globals, while mobile builds pass rnMediaAdapter() (or an equivalent) at construction time. onLocalStream / onRemoteStream are the outputs of the media layer — the engine forwards captured and received streams to the app UI. MediaCallbacks formalizes the same trio (sendMedia, onLocalStream, onRemoteStream) for use inside the engine’s media pipeline.

Mobile Adapter: rnMediaAdapter()

The mobile implementation is backed by react-native-webrtc and lives in streaming/src/lib/media-adapter.ts. It is a factory function returning an object literal that satisfies MediaAdapter:

/** Mobile media adapter backed by react-native-webrtc (casts to the browser-shaped core API) */
export function rnMediaAdapter(): MediaAdapter {
  return {
    createPeerConnection(_roomId: string): RTCPeerConnection {
      // STUN for server-reflexive candidates. TURN was removed pending
      // diagnosis — with the coturn relay added, the remote video forward
      // stopped arriving on the app.
      return new RNPC({ iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] }) as unknown as RTCPeerConnection;
    },
    async getUserMedia(kind: CallKind): Promise<MediaStream> {
      try {
        await ensureMediaPermissions(kind);
        return (await mediaDevices.getUserMedia({ audio: true, video: kind === 'video' })) as unknown as MediaStream;
      } catch (_) {
        try {
          return (await mediaDevices.getUserMedia({ audio: true, video: false })) as unknown as MediaStream;
        } catch (_) {
          return new RNStream() as unknown as MediaStream;
        }
      }
    },
  };
}

Source: media-adapter.ts

Three behaviors stand out:

1. Type casting at the boundary. RNPC, mediaDevices, and RNStream are react-native-webrtc objects; the adapter casts each to the standard DOM types (as unknown as RTCPeerConnection). The engine never sees the native types, and the mobile SDK’s API differences are contained inside this one file.

2. STUN-only ICE configuration. Only stun:stun.l.google.com:19302 is configured. The comment documents a real production incident: adding the coturn TURN relay caused the remote video forward to stop arriving, so TURN was removed pending diagnosis. This is a concrete example of transport policy being an adapter concern — the engine’s signaling logic was unaffected by the change.

3. Two-level fallback chain in getUserMedia. If the requested capture fails (permission denied, no camera, hardware error), the adapter retries with audio-only; if that also fails, it returns an empty RNStream. The call then proceeds with no local media rather than throwing — a deliberate “never fail the call” policy (see Failure Modes).

Android Permission Pre-flight

react-native-webrtc does not request Android 6+ runtime permissions itself, so a naive getUserMedia({ video: true }) throws permission denied, and the adapter’s catch block would silently degrade the call to audio-only — remote users would see a black tile. ensureMediaPermissions() prevents that by requesting grants explicitly before capture:

// Android 6+ requires runtime grants — react-native-webrtc does NOT request
// them itself, so getUserMedia({ video: true }) throws "permission denied" and
// the adapter silently fell back to audio-only → remote users saw a black tile.
async function ensureMediaPermissions(kind: CallKind): Promise<void> {
  if (Platform.OS !== 'android') return;
  const perms: Array<typeof PermissionsAndroid.PERMISSIONS.CAMERA | typeof PermissionsAndroid.PERMISSIONS.RECORD_AUDIO> = [
    PermissionsAndroid.PERMISSIONS.RECORD_AUDIO,
  ];
  if (kind === 'video') perms.push(PermissionsAndroid.PERMISSIONS.CAMERA);
  const results = await PermissionsAndroid.requestMultiple(perms);
  const micOk = results['android.permission.RECORD_AUDIO'] === PermissionsAndroid.RESULTS.GRANTED;
  const camOk = kind !== 'video' || results['android.permission.CAMERA'] === PermissionsAndroid.RESULTS.GRANTED;
  // Throwing makes getUserMedia fall through to its audio-only fallback, so a
  // denied camera degrades the call to audio instead of failing entirely.
  if (!micOk || !camOk) throw new Error('media permission denied');
}

Source: media-adapter.ts

Design intent: RECORD_AUDIO is always requested; CAMERA is only requested for video calls. Throwing a single 'media permission denied' error after the prompts lets the enclosing getUserMedia catch block execute its audio-only fallback — so a user who denies the camera still joins as an audio participant, and a user who denies the microphone gets an empty stream rather than a crash. The pre-flight also guarantees the permission dialog appears before getUserMedia runs, avoiding the black-tile regression described in the comment.

Media URL Resolution: resolveMediaUrl()

Media assets (recorded clips, avatars, attachments) are referenced by relative paths from the API; streaming/src/lib/utils/media.ts converts them to absolute URLs:

import { API_BASE_URL } from '../../config/env';

export function resolveMediaUrl(url?: string | null): string | undefined {
  if (!url) return undefined;
  if (url.startsWith('http://') || url.startsWith('https://')) return url;
  if (url.startsWith('/')) return `${API_BASE_URL}${url}`;
  return `${API_BASE_URL}/${url}`;
}

Source: media.ts

The precedence is: absolute URLs pass through untouched; root-relative paths (/uploads/...) are prefixed with API_BASE_URL; bare relative paths get a / inserted. null/undefined map to undefined, so callers can safely pass optional media fields straight into <img src> / streaming components.

Core Flow

The end-to-end flow of joining a video call on mobile shows how the engine, adapter, permission layer, and capture fallbacks interact:

Step-by-step walkthrough:

1. When the user joins a call, the engine invokes mediaAdapter.getUserMedia(kind) with the call kind ('video' for video calls, 'audio' for audio-only).
2. On Android, the adapter first runs ensureMediaPermissions, which batches RECORD_AUDIO + CAMERA into a single PermissionsAndroid.requestMultiple prompt. On iOS and web this is a no-op (permissions are handled by the OS/react-native-webrtc natively).
3. If any permission is denied, the pre-flight throws; the outer catch retries with video: false — the call degrades to audio. This is the intended UX: a denied camera should not kill the call.
4. If capture itself throws (e.g., no camera hardware), the same fallback runs; a second failure returns an empty MediaStream.
5. The stream is returned to the engine, which forwards it to onLocalStream for the local preview. Remote streams arrive later through onRemoteStream (with participant id and version from the SFU msid).
6. In parallel, the engine establishes the peer session by calling createPeerConnection(roomId), which on mobile constructs a react-native-webrtc RTCPeerConnection with the STUN-only ICE config.

Usage Examples

Wiring the mobile adapter into the engine

On mobile, the adapter is created once and passed through EngineConfig:

import { rnMediaAdapter } from '../lib/media-adapter';

const engine = new RealtimeEngine({
  baseUrl: '/grpc',
  mediaAdapter: rnMediaAdapter(),
  onLocalStream: (stream) => setLocalPreview(stream),
  onRemoteStream: (stream, participantId) => attachRemoteStream(stream, participantId),
});

The engine code is identical on web and mobile; only the mediaAdapter line changes. (Example call-site shape follows the EngineConfig contract above.)

Consuming the contract from the engine side

The engine only ever touches the two factory methods, never platform imports:

const pc: RTCPeerConnection = config.mediaAdapter
  ? config.mediaAdapter.createPeerConnection(roomId)
  : new RTCPeerConnection({ iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] });

const localStream: MediaStream = config.mediaAdapter
  ? await config.mediaAdapter.getUserMedia(kind)
  : await navigator.mediaDevices.getUserMedia({ audio: true, video: kind === 'video' });

Sources: media-adapter.ts, types.ts

Rendering resolved media URLs

import { resolveMediaUrl } from '../lib/utils/media';

// '/uploads/recording-1.mp4'  → `${API_BASE_URL}/uploads/recording-1.mp4`
// 'https://cdn.example.com/x' → unchanged
// null                       → undefined
const src = resolveMediaUrl(recording.url);

Source: media.ts

Configuration Options

EngineConfig (media-related fields)

Source: types.ts

Mobile adapter transport configuration

Source: media-adapter.ts

API Reference

interface MediaAdapter

The platform contract injected into the engine.

createPeerConnection(roomId: string): RTCPeerConnection

Creates (or returns) a peer connection for the given room.

Parameters:

• roomId (string): The room/session identifier. Currently unused by the mobile implementation but reserved for per-room ICE policies.

Returns: A platform RTCPeerConnection (cast from react-native-webrtc on mobile).

Throws: May throw if ICE configuration is invalid; the mobile implementation constructs RNPC with a hardcoded STUN config.

getUserMedia(kind: CallKind): Promise<MediaStream>

Captures local media, degrading gracefully on failure.

Parameters:

• kind ('audio' | 'video'): Whether the camera is required. 'audio' requests mic only; 'video' requests mic + camera.

Returns: Promise<MediaStream> — a full stream, an audio-only stream, or an empty stream, in that order of fallback.

Throws: Never throws in the mobile implementation; all failures are absorbed by the fallback chain. Other implementations may reject (e.g., web without a mic).

rnMediaAdapter(): MediaAdapter

Factory returning the React Native implementation backed by react-native-webrtc.

Returns: A MediaAdapter object literal. Safe to call multiple times; each call creates a fresh adapter with its own permission state.

ensureMediaPermissions(kind: CallKind): Promise<void>

Module-private helper (not exported) that requests Android runtime grants.

Parameters:

• kind (CallKind): Determines whether CAMERA is requested in addition to RECORD_AUDIO.

Throws: Error('media permission denied') when any required permission is not granted; no-op on non-Android platforms.

resolveMediaUrl(url?: string | null): string | undefined

Resolves a media path to an absolute URL using API_BASE_URL.

Parameters:

• url (string | null | undefined): Absolute URL, root-relative path (/x), or bare relative path (x).

Returns: string when a resolvable URL is given; undefined for null/undefined input.

Behavior:

• http:// / https:// prefixes pass through unchanged.
• Leading-/ paths are prefixed with API_BASE_URL as-is.
• Other paths get a / inserted before being prefixed.

Failure Modes, Edge Cases & Concurrency

Camera permission denied (Android). ensureMediaPermissions throws 'media permission denied'; getUserMedia catches it and retries with video: false. Result: the user joins as an audio participant instead of failing the call. This is an explicit design decision documented in the source — a denied camera degrades, never kills, a call.

Microphone permission denied. The same pre-flight throws (mic is always required), and both capture attempts fail, so the adapter returns an empty RNStream. The signaling flow still proceeds; the remote side sees no local media. The engine should surface this state to the UI via onLocalStream (empty stream) so the user knows their mic is off.

Capture hardware failure. Devices without a camera (or with a camera already in use) throw from mediaDevices.getUserMedia; the same audio-only → empty-stream fallback chain applies. Note the empty-stream terminal case is only reached when both capture attempts fail.

TURN relay regression. The source comments document that adding the coturn TURN relay caused remote video to stop arriving on the app; TURN was therefore removed pending diagnosis, leaving STUN-only ICE. Operational implication: hosts behind symmetric NAT may fail to establish media even though signaling succeeds. Anyone re-adding TURN must validate the remote-video forward path end-to-end.

Non-Android platforms. ensureMediaPermissions early-returns unless Platform.OS === 'android', so permission behavior on iOS/web is delegated to the OS / react-native-webrtc.

Concurrency. The adapter holds no mutable shared state: rnMediaAdapter() returns a fresh object per call, and ensureMediaPermissions only awaits requestMultiple (no global locks). Concurrent getUserMedia calls (e.g., rapid re-joins) can trigger multiple permission prompts; the OS serializes them. createPeerConnection does not memoize connections, so the engine must keep its own per-room RTCPeerConnection references — the roomId parameter is currently advisory on mobile.

Performance & Operational Notes

• ICE transport: A single public Google STUN server is used; without TURN, media traversal relies on server-reflexive candidates. NAT-restricted networks may see connection failures — the documented reason TURN was disabled should be re-investigated before production hardening.
• Permission prompt cost: requestMultiple is called on every getUserMedia invocation on Android. Once granted, Android typically returns immediately, but the call is still async; repeated join/leave cycles incur a small per-join overhead. Caching the grant state inside the adapter is a possible optimization (not currently implemented).
• Empty-stream fallback cost: constructing new RNStream() avoids throwing but means downstream consumers must handle a stream with no tracks — UI code should check stream.getTracks().length before attaching.
• URL resolution: resolveMediaUrl is a pure string operation with no caching or network I/O; it is safe to call on every render.

Extension Points

The abstraction is deliberately open for new implementations:

• Desktop / other platforms: implement MediaAdapter (e.g., an Electron adapter using Chromium globals, or a native module adapter) and pass it via EngineConfig.mediaAdapter — no engine changes required.
• Custom ICE policy: swap the STUN/TURN array inside createPeerConnection per deployment; the engine is unaffected.
• Custom capture policy: replace getUserMedia behavior (e.g., select specific camera/mic devices, apply constraints, or inject a test stream) while keeping the CallKind contract.
• Per-room behavior: the roomId parameter of createPeerConnection is currently unused on mobile; implementations may use it for room-specific ICE servers or connection reuse.

Tests

No dedicated test files for the adapter were found in the explored source (streaming/src/lib/media-adapter.ts and streaming/src/lib/utils/media.ts ship without unit tests in the scanned paths). The fallback-chain and permission logic are prime candidates for test coverage; the pure resolveMediaUrl function is trivially testable. Implementation details of test coverage were not found in the source scanned for this page.

Related Links

• Realtime core engine — the consumer of this adapter: call/offer/answer signaling, SFU connection lifecycle, and MediaCallbacks (see realtime-core/src/types.ts for the shared contracts).
• Media cleanup service — backend-side orphan-media storage cleanup: media_cleanup.rs.
• react-native-webrtc — the underlying mobile SDK wrapped by rnMediaAdapter() (permissions and API differences documented in media-adapter.ts).