Mobile Realtime & Media Integration

Mobile Realtime & Media Integration

This page documents how the mobile application integrates realtime signaling with WebRTC media (audio/video calls) through the realtime-core engine — covering the platform-injected MediaAdapter, the single-call MediaManager, the per-call MediaSession WebRTC lifecycle, and the signaling event model that carries SDP and ICE between the client and the SFU.

Purpose and Scope

The realtime-core package provides a transport-agnostic realtime engine (chat messages, presence, call signaling) and a WebRTC media layer. This page focuses on the media integration path for mobile: how the app acquires local media, negotiates a peer connection with the SFU, handles renegotiation offers, buffers early signaling events, and delivers remote streams to the UI — all through a MediaAdapter interface that abstracts the browser (getUserMedia/RTCPeerConnection) from mobile runtimes such as react-native-webrtc.

Sibling concerns intentionally left to other pages:

• Signaling transport details (gRPC-Web wiring, message codec, reconnect policy) live in the signal-client.ts / transport.ts / codec.ts layer of realtime-core; only the media-related signaling messages (media_offer, media_answer, media_ice) are covered here.
• Chat/presence event handling (the RealtimeEvent model) is documented as the contract the media layer uses for signaling, not as a full event reference.
• Server-side SFU behavior is referenced only as the peer endpoint; the client-side negotiation logic is what this page verifies from source.

Overview

The mobile realtime experience has two cooperating planes:

1. Signaling plane — a persistent connection (via SignalClient over a gRPC-Web transport, per EngineConfig.baseUrl) that delivers JSON payloads such as call_offer, media_offer, media_answer, media_ice, and presence events as RealtimeEvent objects.
2. Media plane — WebRTC peer connections negotiated through the signaling plane. The client sends its offer with local SDP, the SFU answers, and further renegotiation offers (e.g., when a peer’s audio/video tracks are added) are answered in sequence. Media packets themselves flow peer-to-peer or through the SFU, outside the signaling channel.

The design goal is platform indirection with a single active call. The EngineConfig.mediaAdapter hook lets the app inject a WebRTC implementation appropriate to the runtime: webMediaAdapter() for browsers, or a custom adapter wrapping react-native-webrtc on mobile. The MediaManager owns exactly one MediaSession at a time (one call at a time), serializes all SDP operations onto a promise chain, and buffers media events that arrive before the session is ready — the three mechanisms that make call setup race-free.

Key concepts:

Architecture

Component roles:

• RealtimeEngine (entry point, engine.ts) — wires the signaling client to the media layer. Its EngineConfig accepts the optional mediaAdapter, onLocalStream, and onRemoteStream hooks that connect media output to the app UI. The baseUrl and tokenProvider options configure the signaling transport.
• SignalClient + Codec + Transport — deliver RealtimeEvent payloads. Media events reaching the engine are routed into MediaManager.handleEvent(signalType, payload); outbound signaling from the session goes back through MediaCallbacks.sendMedia.
• MediaManager — the single coordination point: creates/ends the session, forwards events to it in order, and holds a pending-event buffer for the window between connect() start and session readiness.
• MediaSession — wraps one RTCPeerConnection for a roomId and CallKind. It adds local tracks, installs onicecandidate/ontrack handlers, executes the SDP offer/answer/ICE exchanges, primes keyframes on remote video, and tears the connection down cleanly on close().
• MediaAdapter — injected per platform; webMediaAdapter() uses standard RTCPeerConnection + navigator.mediaDevices.getUserMedia, while mobile builds provide a react-native-webrtc-backed implementation.

The separation matters: the engine and manager never reference navigator or browser WebRTC globals directly (only through the adapter), so the same orchestration code runs on web and mobile. The browser fallback logic in webMediaAdapter (video → audio-only → empty stream) is what keeps call setup from failing outright when a device denies camera access.

Media Layer Deep Dive

Platform Abstraction: MediaAdapter and webMediaAdapter

The MediaAdapter interface is the seam between the engine and the platform WebRTC stack. It exposes exactly two operations: creating a peer connection for a room, and acquiring the local media stream for a call kind.

/** WebRTC media + DataChannels, injected per platform */
export interface MediaAdapter {
  createPeerConnection(roomId: string): RTCPeerConnection;
  getUserMedia(kind: CallKind): Promise<MediaStream>;
}

Source: types.ts

CallKind is the simple union that decides whether video is requested: 'audio' | 'video' (types.ts).

The reference browser implementation, webMediaAdapter(), constructs an RTCPeerConnection with a public STUN server and performs graceful degradation when acquiring media: it first asks for audio + video (for video calls), falls back to audio-only if that fails, and finally returns an empty MediaStream if even audio is unavailable. This guarantees connect() never throws on permission errors; the call proceeds audio-less or stream-less rather than aborting.

/** Browser media adapter: standard RTCPeerConnection + getUserMedia */
export function webMediaAdapter(): MediaAdapter {
  return {
    createPeerConnection(): RTCPickerConnection {
      return new RTCPeerConnection({ iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] });
    },
    async getUserMedia(kind: CallKind): Promise<MediaStream> {
      try {
        return await navigator.mediaDevices.getUserMedia({ audio: true, video: kind === 'video' });
      } catch (_) {
        try {
          return await navigator.mediaDevices.getUserMedia({ audio: true, video: false });
        } catch (_) {
          return new MediaStream();
        }
      }
    },
  };
}

Source: media/adapter.ts

Note: the excerpt above preserves the source verbatim, including the type name RTCPickerConnection as it appears in the file; the actual WebRTC type is RTCPeerConnection.

The factory defaultMediaAdapter() returns the web adapter only when navigator.mediaDevices exists, otherwise null — meaning server-side or non-browser environments get no adapter unless the host injects one (media/adapter.ts). This is the hook a mobile app uses: inject a react-native-webrtc-backed adapter through EngineConfig.mediaAdapter.

Design intent: by keeping getUserMedia inside the adapter, the manager and session stay free of platform globals, and the permission-failure policy (what to do when the camera is denied) is decided in one place per platform instead of scattered through call logic.

Coordination: MediaManager

MediaManager is deliberately small — it owns the single active session, forwards signaling events to it, and serializes SDP operations. Its internal state:

interface PendingMediaEvent {
  signalType: string;
  payload: string;
}

/** Owns the single active media session (one call at a time) */
export class MediaManager {
  private adapter: MediaAdapter;
  private cb: MediaCallbacks;
  private session: MediaSession | null = null;
  private chain: Promise<void> = Promise.resolve();
  private buffer: PendingMediaEvent[] = [];

Source: media/manager.ts

The three mechanisms:

1. Single-session ownership — session is either the one active MediaSession or null; the active getter simply tests session !== null. connect() calls end() first, so starting a new call unconditionally tears down any previous one.
2. Event buffering — handleEvent() with no session pushes { signalType, payload } onto buffer. This covers the race where the SFU pushes a renegotiation offer (the peer’s tracks) while local media is still being acquired.
3. Serialization chain — every SDP operation is appended to this.chain so media_offer/media_answer/media_ice events are processed strictly in arrival order. The chain’s .catch(() => {}) swallows per-event errors so one failed negotiation step cannot stall subsequent events.

async connect(roomId: string, kind: CallKind): Promise<void> {
  this.end();
  const local = await this.adapter.getUserMedia(kind);
  const pc = this.adapter.createPeerConnection(roomId);
  this.session = new MediaSession(roomId, kind, pc, local, this.cb);
  await this.session.sendOffer();
  // The SFU can push a renegotiation offer (peer's tracks) while we were
  // still acquiring local media. Replay any events buffered in that window
  // AFTER our own offer so the caller doesn't permanently miss media and we
  // don't race a remote offer against a pending local offer.
  const pending = this.buffer.splice(0);
  for (const ev of pending) this.handleEvent(ev.signalType, ev.payload);
}

Source: media/manager.ts

connect() ordering matters: local offer first, then replay of buffered remote events. This avoids the classic WebRTC race where a remote offer arrives while a local offer is still being created — answering a remote offer while a local offer is pending corrupts the negotiation state.

handleEvent(signalType: string, payload: string): void {
  const session = this.session;
  if (!session) {
    // Buffer media events that arrive before connect() finishes; replay
    // them once the session exists. Without this, an early renegotiation
    // offer (peer's tracks) is dropped and the caller never receives media.
    this.buffer.push({ signalType, payload });
    return;
  }
  let p: Record<string, unknown>;
  try { p = JSON.parse(payload) as Record<string, unknown>; } catch { return; }
  const sdp = typeof p.sdp === 'string' ? p.sdp : undefined;
  const candidate = typeof p.candidate === 'string' ? p.candidate : undefined;
  // Serialize SDP ops — the SFU sends separate renegotiation offers (audio, video)
  this.chain = this.chain.then(async () => {
    if (signalType === 'media_offer' && sdp) await session.handleOffer(sdp);
    else if (signalType === 'media_answer' && sdp) await session.handleAnswer(sdp);
    else if (signalType === 'media_ice' && candidate) await session.handleIce(candidate);
  }).catch(() => {});
}

Source: media/manager.ts

Malformed payloads are dropped silently (JSON.parse failure returns early). SDP and ICE fields are extracted defensively — an event without a string sdp or candidate simply routes to no branch. end() resets everything: closes the session, nulls the reference, resets the chain, and clears the buffer (media/manager.ts).

Per-Call WebRTC: MediaSession

MediaSession wraps one RTCPeerConnection for a participant in a call room. Its constructor wires local media and the two critical event handlers:

• onicecandidate — each ICE candidate is immediately forwarded to the peer via cb.sendMedia('media_ice', { room_id, candidate }), fire-and-forget with errors swallowed.
• ontrack — remote media arrives here. The handler reconstructs a MediaStream when e.streams[0] is missing (defensive for platforms that deliver tracks without a stream), then, for any newly received video track, requests a keyframe from every video receiver that supports requestKeyFrame. This “belt-and-suspenders” priming makes the picture render immediately instead of waiting for the encoder’s keyframe interval. Finally the stream is surfaced through cb.onRemoteStream(stream, streamId, ++version) — the version counter tracks successive remote streams (e.g., audio then video renegotiation) on the same session.

pc.ontrack = (e) => {
  let s: MediaStream | null = e.streams[0] ?? null;
  if (!s && e.track) {
    const MS: any = (globalThis as any).MediaStream;
    s = MS ? new MS([e.track]) : null;
  }
  console.log('[media] ontrack', {
    trackKind: e.track?.kind,
    hasStreams: !!e.streams?.[0],
    streamId: s ? (s as any).id : undefined,
    trackIds: s ? s.getTracks().map((t) => `${t.kind}:${t.id}`) : [],
    trackState: e.track ? `${e.track.kind}:${e.track.readyState}` : undefined,
  });
  // Ask the origin for a keyframe on any newly-received video track so the
  // picture renders immediately instead of waiting for the encoder's
  // keyframe interval (the SFU also primes this, but belt-and-suspenders).
  if (e.track?.kind === 'video' && this.pc.getReceivers) {
    try {
      this.pc.getReceivers().forEach((r: any) => {
        const track: MediaStreamTrack | undefined = r.track;
        if (track && track.kind === 'video' && (r as any).requestKeyFrame) {
          (r as any).requestKeyFrame();
        }
      });
    } catch { /* requestKeyFrame is optional on some platforms */ }
  }
  if (s) cb.onRemoteStream(s, (s as any).id, ++this.version);
};

Source: media/session.ts

The SDP exchange methods are thin, ordered wrappers over the peer connection state machine:

async sendOffer(): Promise<void> {
  const offer = await this.pc.createOffer();
  await this.pc.setLocalDescription(offer);
  await this.cb.sendMedia('media_offer', { room_id: this.roomId, sdp: offer.sdp ?? '' });
}

async handleOffer(sdp: string): Promise<void> {
  await this.pc.setRemoteDescription({ type: 'offer', sdp });
  const answer = await this.pc.createAnswer();
  await this.pc.setLocalDescription(answer);
  await this.cb.sendMedia('media_answer', { room_id: this.roomId, sdp: answer.sdp ?? '' });
}

async handleAnswer(sdp: string): Promise<void> {
  await this.pc.setRemoteDescription({ type: 'answer', sdp });
}

async handleIce(candidate: string): Promise<void> {
  await this.pc.addIceCandidate({ candidate });
}

Source: media/session.ts

Note that handleOffer (used for SFU-initiated renegotiation) answers every offer — the client is always the answering side for renegotiation, while the initial sendOffer from connect() makes the client the offerer for the first exchange. close() detaches handlers, stops all local sender tracks (s.track?.stop() releases the camera/mic), and closes the peer connection (media/session.ts).

Core Flow: Mobile Call Setup and Media Delivery

The end-to-end flow for a mobile call, as implemented in MediaManager.connect → MediaSession → MediaAdapter:

Step-by-step walkthrough:

1. Teardown first — connect() immediately calls end(), guaranteeing at most one call at a time. Any prior session’s peer connection is closed and its local tracks stopped.
2. Acquire local media — adapter.getUserMedia(kind) may take seconds (permission prompts). The adapter’s fallback chain (video → audio → empty) means this step resolves rather than rejects on permission denial.
3. Create the peer connection — adapter.createPeerConnection(roomId); the room id is not currently used by the web adapter’s constructor but is part of the contract so a mobile adapter could, e.g., configure room-specific ICE servers.
4. Construct the session — local stream is surfaced to the app immediately via cb.onLocalStream, and every local track is added to the connection (addTrack), which implicitly starts ICE candidate gathering.
5. Send the local offer — sendOffer() creates the offer, sets it as the local description, and ships it as a media_offer signaling event.
6. Replay buffered remote events — any media_offer/media_ice that arrived during steps 2–4 is now replayed after the local offer, so a renegotiation offer from the SFU cannot race the pending local offer.
7. Answer renegotiation — a remote media_offer (e.g., the peer’s video track being added) is answered with media_answer; the client is the answering side for all subsequent exchanges.
8. ICE exchange — candidates flow in both directions as media_ice events and are fed to addIceCandidate, serialized on the manager’s promise chain.
9. Remote media delivery — on ontrack, the session builds a stream, requests a keyframe for video (so the first frame renders promptly), and calls onRemoteStream(stream, participantId, version) with the stream id from the SFU’s msid and an incrementing version.

Usage Examples

Wiring the engine on a mobile client

The EngineConfig contract shows how a mobile app injects its WebRTC layer and media callbacks alongside signaling configuration:

export interface EngineConfig {
  /** gRPC-Web base URL, e.g. '/grpc' (Next proxy) or 'http://host:50051' */
  baseUrl: string;
  /** Optional bearer token provider; when omitted, auth is injected by a proxy/cookie */
  tokenProvider?: () => string | Promise<string>;
  /** Stream reconnect delay after drop (ms). Default 3000 */
  reconnectDelayMs?: number;
  onError?: (error: Error) => void;
  /** Injected WebRTC layer (web browser / mobile react-native-webrtc) */
  mediaAdapter?: MediaAdapter;
  onLocalStream?: (stream: MediaStream) => void;
  /** Remote media stream plus the participant id it belongs to (stream.id from the SFU msid). */
  onRemoteStream?: (stream: MediaStream, participantId?: string, version?: number) => void;
}

Source: types.ts

A mobile build would supply a react-native-webrtc-backed object satisfying MediaAdapter (same shape as webMediaAdapter), pass it as mediaAdapter, and use onRemoteStream to attach the incoming stream to a <RTCView>/video element.

Media callbacks contract

The engine’s media layer communicates back through MediaCallbacks — the same three operations used by the session and manager:

export interface MediaCallbacks {
  sendMedia: (signalType: string, payload: Record<string, unknown>) => Promise<void>;
  onLocalStream: (stream: MediaStream) => void;
  onRemoteStream: (stream: MediaStream, participantId?: string, version?: number) => void;
}

Source: types.ts

sendMedia is the outbound signaling channel: it is called with 'media_offer', 'media_answer', or 'media_ice' plus a payload carrying room_id and either sdp or candidate. The engine is responsible for routing that onto the signaling transport (alongside call-control events such as call_offer/call_answer, represented by SendSignalInput in types.ts).

Placing a call programmatically

// Pseudocode assembled from the public surface of the media layer:
const manager = new MediaManager(
  defaultMediaAdapter() ?? myMobileAdapter,
  {
    sendMedia: (signalType, payload) => signalClient.sendSignal({ targetId: peerId, signalType, payload: JSON.stringify(payload) }),
    onLocalStream: (stream) => previewView.attachStream(stream),
    onRemoteStream: (stream, participantId, version) => remoteView.attachStream(stream),
  },
);

await manager.connect(roomId, 'video');   // sends media_offer
manager.handleEvent('media_ice', JSON.stringify({ candidate: 'candidate:...' }));
manager.end();                            // hang up: closes PC, stops local tracks

The three calls shown mirror the exact public API implemented in media/manager.ts: connect(roomId, kind), handleEvent(signalType, payload), and end().

Configuration Options

The media integration is configured entirely through EngineConfig (the media layer itself takes its dependencies via constructor injection rather than global configuration). Options relevant to realtime + media integration:

Source: types.ts

Two fixed (non-configurable) constants live in code: the STUN server stun:stun.l.google.com:19302 used by webMediaAdapter (media/adapter.ts), and the media signal types media_offer, media_answer, media_ice dispatched by MediaManager.handleEvent (media/manager.ts).

API Reference

MediaAdapter (interface)

createPeerConnection(roomId: string): RTCPeerConnection;
getUserMedia(kind: CallKind): Promise<MediaStream>;

• createPeerConnection(roomId) — Returns: a new RTCPeerConnection. The roomId is passed so platform adapters can select room-specific configuration (ICE servers, constraints). The web adapter ignores it beyond construction.
• getUserMedia(kind) — Returns: a Promise<MediaStream>; kind === 'video' requests video, 'audio' requests audio only. The web implementation degrades to audio-only then to an empty stream on failure; it throws only if no navigator.mediaDevices exists (guarded by defaultMediaAdapter).

MediaManager (class)

MediaSession (class)

MediaCallbacks (interface)

Failure Modes, Edge Cases & Concurrency

The media layer’s robustness comes from explicit handling of several race conditions and platform quirks, all visible in the source:

Early renegotiation offer vs. local offer race. The SFU can push a media_offer (peer’s tracks) while connect() is still awaiting getUserMedia. If dropped, the caller would never receive media; if processed immediately, the remote offer would race the pending local offer and corrupt negotiation. The fix is two-sided: events arriving before the session exists are buffered (MediaManager.buffer), and buffered events are replayed only after sendOffer() completes (media/manager.ts).

SDP operation ordering. handleEvent appends every operation to a single promise chain (this.chain = this.chain.then(...)), so concurrent media_offer/media_answer/media_ice events — which the SFU may emit as separate messages for audio and video renegotiation — are applied strictly in arrival order. .catch(() => {}) on the chain ensures one failed step (e.g., an ICE candidate that no longer applies) does not deadlock the rest.

Media permission denial. webMediaAdapter.getUserMedia degrades video → audio-only → empty MediaStream instead of throwing. A call therefore proceeds without camera, without microphone, or with no local media at all, and the app is informed via onLocalStream(emptyStream) rather than an exception. The trade-off: call quality degrades silently unless the app inspects the stream’s tracks.

Malformed signaling payloads. handleEvent wraps JSON.parse in try/catch and drops unparseable payloads; events lacking a string sdp or candidate match no branch and are ignored. This makes the media layer tolerant of transport-level corruption without breaking the serialization chain.

Missing streams on ontrack. Some platforms deliver tracks without an associated MediaStream (e.streams[0] undefined). The session reconstructs a stream from the track via globalThis.MediaStream and only invokes onRemoteStream when a usable stream exists — otherwise the track is logged but not surfaced.

Optional keyframe API. requestKeyFrame is guarded by both a feature check (this.pc.getReceivers) and a per-receiver capability check, wrapped in try/catch, because the API is not available on every platform. This keeps video-arrival priming from ever throwing.

Concurrency model. Concurrency is deliberately serialized rather than parallel: one session at a time (enforced by connect() calling end()), and one SDP operation at a time (enforced by the promise chain). The only asynchronous overlap is local media acquisition, which is why the buffer exists. There is no locking mechanism beyond this ordering — callers must not call connect() concurrently (the second call would simply end the first).

Teardown hygiene. MediaSession.close() nulls both event handlers before closing the peer connection and stops every local sender track, releasing the camera/mic; MediaManager.end() then clears the chain and buffer so a new call starts from a clean state.

Performance & Operational Considerations

• Keyframe priming on remote video — the session requests a keyframe from video receivers on every ontrack, reducing first-paint latency for the incoming picture. This is a client-side complement to SFU-side priming; both are “belt-and-suspenders” per the source comment.
• Fire-and-forget ICE — outbound media_ice is sent with .catch(() => {}), trading error visibility for signaling latency; ICE candidates are time-sensitive and must not be queued behind retries.
• Single-call constraint — the manager’s “one call at a time” model bounds resource usage to one peer connection and one set of local tracks, which is the right trade-off for mobile memory/battery. Multi-call scenarios are not supported by design.
• Reconnect behavior — transport-level reconnects are governed by EngineConfig.reconnectDelayMs (default 3000 ms); the media layer itself does not re-negotiate after a signaling drop — a reconnect requires the app to re-drive connect().

Extension Points

• MediaAdapter injection — the primary extension seam. A mobile app supplies a react-native-webrtc-backed implementation of createPeerConnection + getUserMedia (mirroring webMediaAdapter) via EngineConfig.mediaAdapter; the manager and session are platform-agnostic.
• EngineConfig callbacks — onLocalStream, onRemoteStream, and onError let the app bind media to UI (preview/remote views) and observe failures without subclassing.
• MediaCallbacks.sendMedia — the engine routes outbound media signaling; an app can observe or log signaling by wrapping this callback.
• defaultMediaAdapter — returns null outside browsers, so server-side or non-DOM environments cleanly fall back to adapter-less operation rather than crashing on navigator access.

Tests

No dedicated test files for the media layer were found in this repository (realtime-core currently ships source plus build configuration: package.json, tsconfig.json, bun.lock, and src/**). The observable invariants that tests should cover, based on the implementation, are: buffered-event replay ordering in MediaManager.connect, serialization of mixed offer/answer/ICE sequences on the promise chain, fallback behavior of webMediaAdapter.getUserMedia, and close() releasing local tracks. Implementation details of test coverage are not available in the repository at this time.

Related Links

• realtime-core/src/media/adapter.ts — platform adapter factory (webMediaAdapter, defaultMediaAdapter)
• realtime-core/src/media/manager.ts — MediaManager: single-call ownership, event buffering, SDP serialization
• realtime-core/src/media/session.ts — MediaSession: offer/answer/ICE lifecycle, keyframe priming, teardown
• realtime-core/src/types.ts — EngineConfig, MediaAdapter, MediaCallbacks, RealtimeEvent, SendSignalInput
• realtime-core/src/engine.ts — engine entry point wiring signaling and media (signaling details on the realtime signaling page)
• realtime-core/src/signal-client.ts — signaling client transport (gRPC-Web), covered by the signaling/transport catalog page