# 08b — LiveKit Streaming: Vollständige Implementierungsanleitung > **Ziel:** Echtzeit Audio/Video-Calls und Screen-Sharing in Pyramid via LiveKit SFU, > authentifiziert über MatrixRTC (Matrix-Standard), selbst gehostet auf dem Raspberry Pi. --- ## Inhaltsverzeichnis - [Architektur-Überblick](#architektur-überblick) - [Selbst-gehostete Infrastruktur](#selbst-gehostete-infrastruktur) - [LiveKit-Server Setup (Raspberry Pi)](#livekit-server-setup-raspberry-pi) - [LiveKit JWT Service (lk-jwt-service)](#livekit-jwt-service-lk-jwt-service) - [Synapse MatrixRTC Konfiguration](#synapse-matrixrtc-konfiguration) - [TURN-Server Setup (coturn)](#turn-server-setup-coturn) - [Nginx Reverse-Proxy](#nginx-reverse-proxy) - [Flutter: LiveKit-Paket & Abhängigkeiten](#flutter-livekit-paket--abhängigkeiten) - [MatrixRTC Membership Flow (Signaling)](#matrixrtc-membership-flow-signaling) - [LiveKit-Token holen (OpenID-Austausch)](#livekit-token-holen-openid-austausch) - [LiveKit Room verbinden](#livekit-room-verbinden) - [Audio-Call implementieren](#audio-call-implementieren) - [Video-Call implementieren](#video-call-implementieren) - [Screen-Sharing implementieren](#screen-sharing-implementieren) - [Klingel-System (Ringing)](#klingel-system-ringing) - [Call-State-Management (Riverpod)](#call-state-management-riverpod) - [Call-UI Komponenten](#call-ui-komponenten) - [Hintergrund-Service (Android)](#hintergrund-service-android) - [Komplettes Dart-Implementierungs-Template](#komplettes-dart-implementierungs-template) - [Troubleshooting & Diagnose](#troubleshooting--diagnose) --- ## Architektur-Überblick ``` ┌─────────────────────────────────────────────────────────────────┐ │ Pyramid App (Flutter) │ │ │ │ MatrixRTC Signaling LiveKit Media │ │ ┌──────────────────┐ ┌─────────────────────┐ │ │ │ Matrix SDK │ │ livekit_client pkg │ │ │ │ - call.member │ │ - Audio/Video tracks │ │ │ │ state events │ │ - Screen share │ │ │ │ - to-device msgs │ │ - Data channels │ │ │ └──────┬───────────┘ └──────────┬──────────┘ │ │ │ │ │ └─────────┼────────────────────────────────┼──────────────────────┘ │ HTTPS/WSS │ WSS (WebSocket Secure) ▼ ▼ ┌─────────────────────┐ ┌──────────────────────────────┐ │ Synapse (Matrix HS)│ │ LiveKit Server (SFU) │ │ matrix.steggi- │ │ livekit.steggi-matrix.work │ │ matrix.work │ │ Port: 7880 (HTTPS/WS) │ │ │ │ Port: 7881 (TURN/TLS) │ │ OpenID Token API │ │ Port: 50100-50200 (UDP RTP) │ └────────┬────────────┘ └───────────────┬──────────────┘ │ OpenID Token │ ▼ │ ┌────────────────────┐ │ │ lk-jwt-service │───────── JWT Token ───────┘ │ (Token Exchange) │ │ Port: 8080 │ └────────────────────┘ │ ▼ ┌────────────────────┐ │ coturn (TURN) │ │ Port: 3478 UDP │ │ Port: 5349 TLS │ └────────────────────┘ Nginx Reverse Proxy (Port 443): /livekit/ → LiveKit :7880 /lk-jwt/ → lk-jwt-service :8080 (coturn läuft direkt auf 3478/5349) ``` --- ## Selbst-gehostete Infrastruktur ### Benötigte Komponenten | Dienst | Zweck | Port | Software | |--------|-------|------|----------| | **LiveKit Server** | SFU — verteilt Media-Streams | 7880 (WS/HTTP), 7881 (TURN/TLS), 50100-50200 UDP | `livekit/livekit-server` | | **lk-jwt-service** | Token-Austausch OpenID → LiveKit JWT | 8080 | `element-hq/lk-jwt-service` | | **coturn** | TURN-Server für NAT-Traversal | 3478 UDP, 5349 TLS | `coturn` | | **Nginx** | Reverse Proxy + SSL-Termination | 443 | `nginx` | | **Synapse** | Matrix Homeserver (bereits vorhanden) | 8448 | ✓ läuft | ### DNS-Einträge (Beispiel für steggi-matrix.work) ``` livekit.steggi-matrix.work A turn.steggi-matrix.work A ``` --- ## LiveKit-Server Setup (Raspberry Pi) ### Installation ```bash # LiveKit Binary herunterladen (ARM64 für Raspberry Pi 4/5) curl -sSL https://get.livekit.io | bash # oder manuell: wget https://github.com/livekit/livekit/releases/latest/download/livekit_linux_arm64.tar.gz tar xzf livekit_linux_arm64.tar.gz sudo mv livekit-server /usr/local/bin/ # API-Key und Secret generieren livekit-server generate-keys # Ausgabe: # API Key: APIxxxxxxxxxxxxxxxxxx # API Secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx # → SICHER AUFBEWAHREN! ``` ### Konfiguration: `/etc/livekit/livekit.yaml` ```yaml port: 7880 bind_addresses: - "" # Alle Interfaces # API-Schlüssel (aus generate-keys) keys: APIxxxxxxxxxxxxxxxxxx: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx # WebRTC-Einstellungen rtc: tcp_port: 7881 # TURN over TLS port_range_start: 50100 port_range_end: 50200 # Öffentliche IP des Raspberry Pi (wichtig für NAT!) use_external_ip: true # STUN-Server für ICE (externe Erreichbarkeit prüfen) stun_servers: - stun:stun.l.google.com:19302 # TURN eingebaut (LiveKit kann selbst TURN machen) turn: enabled: true domain: livekit.steggi-matrix.work tls_port: 5349 # TURN über TLS udp_port: 3478 # Logging logging: level: info pion_level: error # WebRTC-interne Logs reduzieren # Redis (optional, für Cluster — beim Pi nicht nötig) # redis: # address: localhost:6379 ``` ### Systemd Service: `/etc/systemd/system/livekit.service` ```ini [Unit] Description=LiveKit SFU Server After=network.target Wants=network-online.target [Service] Type=simple User=livekit Group=livekit ExecStart=/usr/local/bin/livekit-server --config /etc/livekit/livekit.yaml Restart=on-failure RestartSec=5 LimitNOFILE=65536 # Logs StandardOutput=journal StandardError=journal [Install] WantedBy=multi-user.target ``` ```bash # User anlegen und Service aktivieren sudo useradd -r -s /bin/false livekit sudo systemctl daemon-reload sudo systemctl enable livekit sudo systemctl start livekit sudo systemctl status livekit # Logs verfolgen sudo journalctl -u livekit -f ``` --- ## LiveKit JWT Service (lk-jwt-service) Dieser Service tauscht Matrix OpenID-Tokens gegen LiveKit JWTs aus. Das ist die **Brücke zwischen Matrix-Authentifizierung und LiveKit**. ### Installation ```bash # Go installieren (falls nicht vorhanden) # Für Raspberry Pi (ARM64): wget https://go.dev/dl/go1.22.0.linux-arm64.tar.gz sudo tar -C /usr/local -xzf go1.22.0.linux-arm64.tar.gz export PATH=$PATH:/usr/local/go/bin # lk-jwt-service bauen git clone https://github.com/element-hq/lk-jwt-service cd lk-jwt-service go build -o lk-jwt-service . sudo mv lk-jwt-service /usr/local/bin/ ``` ### Konfiguration: `/etc/lk-jwt-service/config.yaml` ```yaml # LiveKit-Verbindung livekit_url: wss://livekit.steggi-matrix.work livekit_key: APIxxxxxxxxxxxxxxxxxx # selber Key wie in livekit.yaml livekit_secret: xxxxxxxxxxxxxxxxxxxxx # selbes Secret # Matrix-Homeserver für OpenID-Validierung matrix_server: https://matrix.steggi-matrix.work # Welche Matrix-Homeserver dürfen den Service nutzen allowed_matrix_servers: - steggi-matrix.work # Server-Port port: 8080 ``` ### Systemd Service: `/etc/systemd/system/lk-jwt-service.service` ```ini [Unit] Description=LiveKit JWT Token Service for MatrixRTC After=network.target livekit.service [Service] Type=simple User=livekit ExecStart=/usr/local/bin/lk-jwt-service --config /etc/lk-jwt-service/config.yaml Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target ``` ```bash sudo systemctl enable lk-jwt-service sudo systemctl start lk-jwt-service ``` ### API-Endpoint des lk-jwt-service ``` POST /sfu/get Content-Type: application/json { "room": "!roomId:server", "openid_token": { "access_token": "...", ← von client.requestOpenIdToken() "token_type": "Bearer", "matrix_server_name": "steggi-matrix.work" }, "device_id": "PYRAMID_DEVICE_ID" } Antwort: { "url": "wss://livekit.steggi-matrix.work", "jwt": "eyJ..." ← LiveKit JWT Token } ``` --- ## Synapse MatrixRTC Konfiguration In der Synapse-Konfiguration (`homeserver.yaml`): ```yaml # MatrixRTC / LiveKit Integration experimental_features: msc3401_enabled: true # MatrixRTC Group Calls msc4140_enabled: true # Delayed events (für Membership-Cleanup) # Erlaubt das Senden von com.famedly.call.member State-Events rc_message: per_second: 100 burst_count: 200 # LiveKit-Service URL im .well-known bekannt machen (MSC4143) # → client kann automatisch den richtigen JWT-Service finden ``` ### `.well-known/matrix/client` erweitern Nginx liefert diese JSON-Datei. Ergänze LiveKit-Info: ```json { "m.homeserver": { "base_url": "https://matrix.steggi-matrix.work" }, "org.matrix.msc4143.rtc_foci": [ { "type": "livekit", "livekit_service_url": "https://livekit.steggi-matrix.work/lk-jwt" } ] } ``` --- ## TURN-Server Setup (coturn) LiveKit hat einen eingebauten TURN-Server, aber coturn als Fallback ist empfohlen. ```bash sudo apt install coturn -y ``` ### `/etc/turnserver.conf` ```conf # Listening listening-port=3478 tls-listening-port=5349 listening-ip=0.0.0.0 # Externe IP (Raspberry Pi öffentliche IP) external-ip= # Realm realm=turn.steggi-matrix.work server-name=turn.steggi-matrix.work # TLS-Zertifikate (Let's Encrypt) cert=/etc/letsencrypt/live/turn.steggi-matrix.work/fullchain.pem pkey=/etc/letsencrypt/live/turn.steggi-matrix.work/privkey.pem # Auth (shared secret für Matrix) use-auth-secret static-auth-secret=ZUFAELLIGES_LANGES_GEHEIMNIS_HIER # Relay-Ports min-port=49152 max-port=65535 # Sicherheit no-multicast-peers no-loopback-peers denied-peer-ip=10.0.0.0-10.255.255.255 denied-peer-ip=192.168.0.0-192.168.255.255 denied-peer-ip=172.16.0.0-172.31.255.255 # Logging log-file=/var/log/turnserver.log ``` ```bash # TURN-Service aktivieren sudo systemctl enable coturn sudo systemctl start coturn # Synapse: TURN-Credentials bekannt machen (in homeserver.yaml) turn_uris: - "turn:turn.steggi-matrix.work:3478?transport=udp" - "turn:turn.steggi-matrix.work:3478?transport=tcp" - "turns:turn.steggi-matrix.work:5349?transport=tcp" turn_shared_secret: "ZUFAELLIGES_LANGES_GEHEIMNIS_HIER" turn_user_lifetime: 86400000 turn_allow_guests: false ``` --- ## Nginx Reverse-Proxy ```nginx # /etc/nginx/sites-available/livekit.steggi-matrix.work server { listen 443 ssl http2; server_name livekit.steggi-matrix.work; ssl_certificate /etc/letsencrypt/live/livekit.steggi-matrix.work/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/livekit.steggi-matrix.work/privkey.pem; # LiveKit WebSocket + HTTP API location / { proxy_pass http://localhost:7880; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 3600s; proxy_send_timeout 3600s; proxy_buffering off; client_max_body_size 0; } # lk-jwt-service (Token-Austausch) location /lk-jwt/ { rewrite ^/lk-jwt/(.*) /$1 break; proxy_pass http://localhost:8080; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; add_header 'Access-Control-Allow-Origin' '*' always; add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always; add_header 'Access-Control-Allow-Headers' 'Authorization, Content-Type' always; } } # HTTP → HTTPS Redirect server { listen 80; server_name livekit.steggi-matrix.work; return 301 https://$host$request_uri; } ``` ```bash sudo nginx -t && sudo nginx -s reload ``` ### Firewall-Regeln ```bash # UFW sudo ufw allow 443/tcp # HTTPS (LiveKit + lk-jwt) sudo ufw allow 3478/udp # TURN UDP sudo ufw allow 3478/tcp # TURN TCP sudo ufw allow 5349/tcp # TURNS TLS sudo ufw allow 7881/tcp # LiveKit TURN/TLS (eingebaut) sudo ufw allow 50100:50200/udp # RTP Media Ports # Für Raspberry Pi hinter Router: Port-Forwarding nötig! # Im Router: alle oben genannten Ports → Pi-interne IP weiterleiten ``` --- ## Flutter: LiveKit-Paket & Abhängigkeiten ### `pubspec.yaml` ```yaml dependencies: livekit_client: ^2.4.0 # LiveKit Flutter SDK flutter_webrtc: ^0.10.0 # WebRTC (wird von livekit_client benötigt) permission_handler: ^11.0.0 # Mikrofon/Kamera-Berechtigung wakelock_plus: ^1.0.0 # Screen während Call aktiv halten # Bereits vorhanden: matrix: ^6.x.x flutter_riverpod: ^2.x.x ``` ### Android: `android/app/src/main/AndroidManifest.xml` ```xml ``` ### iOS: `ios/Runner/Info.plist` ```xml NSCameraUsageDescription Pyramid benötigt die Kamera für Video-Calls NSMicrophoneUsageDescription Pyramid benötigt das Mikrofon für Audio-Calls ``` --- ## MatrixRTC Membership Flow (Signaling) MatrixRTC verwendet State-Events im Raum um anzuzeigen, wer im Call ist. ```dart // lib/features/call/matrixrtc_service.dart const kCallMemberEventType = 'org.matrix.msc3401.call.member'; class MatrixRtcMembership { final String roomId; final String deviceId; final String callId; final String livekitServiceUrl; MatrixRtcMembership({ required this.roomId, required this.deviceId, required this.callId, required this.livekitServiceUrl, }); // ── Call beitreten: Membership-Event senden ──────────────────────────── Future join(Client matrixClient) async { final expiresTs = DateTime.now() .add(const Duration(hours: 4)) .millisecondsSinceEpoch; await matrixClient.sendState( roomId, kCallMemberEventType, { 'application': 'm.call', 'call_id': callId, 'device_id': deviceId, 'expires_ts': expiresTs, 'membershipID': '${deviceId}_${DateTime.now().millisecondsSinceEpoch}', 'backend': { 'type': 'livekit', 'livekit_service_url': livekitServiceUrl, }, }, stateKey: '$deviceId:$callId', ); } // ── Call verlassen: Membership entfernen ─────────────────────────────── Future leave(Client matrixClient) async { await matrixClient.sendState( roomId, kCallMemberEventType, {}, // Leer = Austritt stateKey: '$deviceId:$callId', ); } // ── Aktive Mitglieder auflisten ─────────────────────────────────────── static List> getActiveMembers(Room room, String callId) { final states = room.states[kCallMemberEventType] ?? {}; final now = DateTime.now().millisecondsSinceEpoch; return states.values .where((e) { final content = e.content; return content['call_id'] == callId && (content['expires_ts'] as int? ?? 0) > now && content.isNotEmpty; }) .map((e) => Map.from(e.content)) .toList(); } // ── Auf Membership-Änderungen hören ────────────────────────────────── static Stream watchCallMembers(Client client, String roomId) { return client.onSync.stream .where((update) { final rooms = update.rooms?.join; if (rooms == null) return false; final roomUpdate = rooms[roomId]; return roomUpdate?.state?.events?.any( (e) => e.type == kCallMemberEventType) ?? false; }) .map((_) {}); } } ``` --- ## LiveKit-Token holen (OpenID-Austausch) ```dart // lib/features/call/livekit_service.dart import 'dart:convert'; import 'package:http/http.dart' as http; import 'package:matrix/matrix.dart'; class LiveKitService { static const String _jwtServicePath = '/lk-jwt/sfu/get'; /// Tauscht ein Matrix OpenID-Token gegen einen LiveKit JWT. /// [livekitServiceUrl] kommt aus dem MatrixRTC-Membership-Event des Raums. static Future getToken({ required Client matrixClient, required String roomId, required String livekitServiceUrl, }) async { // 1. OpenID-Token vom Homeserver holen final openIdToken = await matrixClient.requestOpenIdToken( matrixClient.userID!, ); // 2. JWT beim lk-jwt-service tauschen final serviceUrl = Uri.parse(livekitServiceUrl).resolve(_jwtServicePath); final response = await http.post( serviceUrl, headers: {'Content-Type': 'application/json'}, body: jsonEncode({ 'room': roomId, 'openid_token': { 'access_token': openIdToken.accessToken, 'token_type': 'Bearer', 'matrix_server_name': openIdToken.matrixServerName, }, 'device_id': matrixClient.deviceID, }), ).timeout(const Duration(seconds: 10)); if (response.statusCode != 200) { throw Exception( 'lk-jwt-service Fehler ${response.statusCode}: ${response.body}'); } final data = jsonDecode(response.body) as Map; return LiveKitCredentials( url: data['url'] as String, jwt: data['jwt'] as String, ); } } class LiveKitCredentials { final String url; // wss://livekit.steggi-matrix.work final String jwt; // LiveKit JWT Token const LiveKitCredentials({required this.url, required this.jwt}); } ``` --- ## LiveKit Room verbinden ```dart // lib/features/call/livekit_room_manager.dart import 'package:livekit_client/livekit_client.dart'; class LiveKitRoomManager { Room? _room; EventsListener? _listener; Room get room => _room!; bool get isConnected => _room?.connectionState == ConnectionState.connected; Future connect({ required String url, required String token, bool audioEnabled = true, bool videoEnabled = false, }) async { _room = Room( roomOptions: RoomOptions( adaptiveStream: true, // Qualität automatisch anpassen dynacast: true, // Inaktive Streams pausieren defaultAudioPublishOptions: const AudioPublishOptions( name: 'microphone', audioBitrate: 24000, // 24 kbps — gut für Sprache ), defaultVideoPublishOptions: VideoPublishOptions( simulcast: true, // Mehrere Qualitätsstufen senden videoEncoding: VideoEncoding( maxBitrate: 1200 * 1000, // 1.2 Mbps max maxFramerate: 30, ), ), defaultScreenShareCaptureOptions: const ScreenShareCaptureOptions( useiOSBroadcastExtension: true, params: VideoParameters( dimensions: VideoDimensions(1920, 1080), encoding: VideoEncoding(maxBitrate: 3 * 1000 * 1000, maxFramerate: 15), ), ), defaultCameraCaptureOptions: CameraCaptureOptions( cameraPosition: CameraPosition.front, params: VideoParameters( dimensions: VideoDimensions(1280, 720), encoding: VideoEncoding(maxBitrate: 1200 * 1000, maxFramerate: 30), ), ), ), ); _listener = _room!.createListener(); try { await _room!.connect(url, token, connectOptions: const ConnectOptions( autoSubscribe: true, // Alle Tracks automatisch empfangen rtcConfig: RTCConfiguration( iceTransportPolicy: RTCIceTransportPolicy.all, ), ), ); // Mikrofon aktivieren if (audioEnabled) { await _room!.localParticipant?.setMicrophoneEnabled(true); } // Kamera aktivieren if (videoEnabled) { await _room!.localParticipant?.setCameraEnabled(true); } } catch (e) { await _room?.dispose(); _room = null; rethrow; } } Future disconnect() async { await _room?.localParticipant?.setMicrophoneEnabled(false); await _room?.localParticipant?.setCameraEnabled(false); await _room?.disconnect(); await _room?.dispose(); _room = null; } // Event-Listener void onParticipantConnected(void Function(RemoteParticipant) callback) { _listener?.on( (event) => callback(event.participant)); } void onParticipantDisconnected(void Function(RemoteParticipant) callback) { _listener?.on( (event) => callback(event.participant)); } void onTrackPublished( void Function(RemoteParticipant, RemoteTrackPublication) callback) { _listener?.on((event) => callback(event.participant, event.publication)); } void onTrackSubscribed( void Function(Track, TrackPublication, RemoteParticipant) callback) { _listener?.on( (event) => callback(event.track, event.publication, event.participant)); } } ``` --- ## Audio-Call implementieren ```dart // lib/features/call/call_notifier.dart (Audio-Call) import 'package:permission_handler/permission_handler.dart'; Future startAudioCall({ required Client matrixClient, required String roomId, }) async { // 1. Berechtigung anfordern final status = await Permission.microphone.request(); if (!status.isGranted) throw Exception('Mikrofon-Berechtigung verweigert'); // 2. LiveKit-Service-URL aus Raum-Metadaten holen (oder Default) const livekitServiceUrl = 'https://livekit.steggi-matrix.work'; // 3. MatrixRTC Membership senden final callId = roomId; // Convention: callId = roomId final membership = MatrixRtcMembership( roomId: roomId, deviceId: matrixClient.deviceID!, callId: callId, livekitServiceUrl: livekitServiceUrl, ); await membership.join(matrixClient); // 4. LiveKit-Token holen final credentials = await LiveKitService.getToken( matrixClient: matrixClient, roomId: roomId, livekitServiceUrl: livekitServiceUrl, ); // 5. Mit LiveKit verbinden (nur Audio) final manager = LiveKitRoomManager(); await manager.connect( url: credentials.url, token: credentials.jwt, audioEnabled: true, videoEnabled: false, ); // 6. Screen aktiv halten WakelockPlus.enable(); } // Stummschalten Future setMuted(LiveKitRoomManager manager, bool muted) async { await manager.room.localParticipant?.setMicrophoneEnabled(!muted); } // Lautsprecher umschalten (Android) Future setSpeakerOn(bool speakerOn) async { await Helper.setSpeakerphoneOn(speakerOn); } ``` --- ## Video-Call implementieren ```dart // Widget: Lokale Kamera-Vorschau class LocalVideoView extends StatelessWidget { final LocalParticipant participant; const LocalVideoView({required this.participant, super.key}); @override Widget build(BuildContext context) { return StreamBuilder( stream: participant.trackPublicationsNotifier, builder: (ctx, _) { final videoPub = participant.videoTrackPublications .where((p) => p.source == TrackSource.camera) .firstOrNull; if (videoPub?.track case final VideoTrack track) { return VideoTrackRenderer(track); } return const SizedBox(); }, ); } } // Widget: Remote-Teilnehmer Video class RemoteVideoView extends StatelessWidget { final RemoteParticipant participant; const RemoteVideoView({required this.participant, super.key}); @override Widget build(BuildContext context) { return StreamBuilder( stream: participant.trackPublicationsNotifier, builder: (ctx, _) { final videoPub = participant.videoTrackPublications .where((p) => p.source == TrackSource.camera || p.source == TrackSource.screenShareVideo) .firstOrNull; if (videoPub?.track case final VideoTrack track) { return VideoTrackRenderer(track, fit: RTCVideoViewObjectFit.contain); } return Container( color: Colors.black87, child: const Center( child: Icon(Icons.person, size: 80, color: Colors.white38), ), ); }, ); } } // Kamera ein-/ausschalten Future toggleCamera(Room room) async { final enabled = room.localParticipant?.isCameraEnabled() ?? false; await room.localParticipant?.setCameraEnabled(!enabled); } // Kamera wechseln (Vorder-/Rückkamera) Future switchCamera(Room room) async { final cameraPub = room.localParticipant?.videoTrackPublications .where((p) => p.source == TrackSource.camera) .firstOrNull; if (cameraPub?.track case final LocalVideoTrack track) { await track.switchCamera(); } } ``` --- ## Screen-Sharing implementieren ```dart // Android: benötigt MediaProjection-Foreground-Service // Screen-Sharing starten Future startScreenShare(Room room) async { // Android: MediaProjection-Berechtigung anfordern (System-Dialog) await room.localParticipant?.setScreenShareEnabled( true, captureScreenAudio: true, // Audio mitschneiden ); } // Screen-Sharing beenden Future stopScreenShare(Room room) async { await room.localParticipant?.setScreenShareEnabled(false); } // Android Foreground Service für Screen-Sharing // android/app/src/main/kotlin/chat/pyramid/pyramid/ScreenShareService.kt: /* class ScreenShareService : Service() { override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int { val notification = NotificationCompat.Builder(this, "call_channel") .setSmallIcon(R.drawable.ic_notification) .setContentTitle("Bildschirm wird geteilt") .setContentText("Pyramid teilt deinen Bildschirm") .build() startForeground(NOTIF_ID, notification, ServiceInfo.FOREGROUND_SERVICE_TYPE_MEDIA_PROJECTION) return START_STICKY } companion object { const val NOTIF_ID = 2001 } } */ ``` --- ## Klingel-System (Ringing) ```dart // lib/features/call/ringing_service.dart // ── Anruf ankündigen (to-device Event) ───────────────────────────────── Future ringUser({ required Client matrixClient, required String targetUserId, required String roomId, required String callId, }) async { await matrixClient.sendToDevice( 'com.famedly.call', // Event-Typ (wie element/fluffychat) { targetUserId: { '*': { // Alle Geräte des Users 'call_id': callId, 'room_id': roomId, 'type': 'invite', 'sender_session_id': matrixClient.deviceID, 'sender_name': matrixClient.displayName, 'version': 1, }, }, }, ); } // ── Klingeln abbrechen ──────────────────────────────────────────────── Future cancelRing({ required Client matrixClient, required String targetUserId, required String callId, }) async { await matrixClient.sendToDevice( 'com.famedly.call', { targetUserId: { '*': {'call_id': callId, 'type': 'hangup', 'reason': 'user_hangup'}, }, }, ); } // ── Eingehende to-device Events abhören ────────────────────────────── StreamSubscription listenForIncomingCalls( Client matrixClient, void Function(IncomingCall) onCall) { return matrixClient.onToDeviceEvent.stream .where((e) => e.type == 'com.famedly.call') .listen((event) { final type = event.content['type'] as String?; final callId = event.content['call_id'] as String?; final roomId = event.content['room_id'] as String?; if (type == 'invite' && callId != null && roomId != null) { onCall(IncomingCall( callId: callId, roomId: roomId, callerId: event.senderId, senderName: event.content['sender_name'] as String?, )); } if (type == 'hangup') { // Klingeln beenden (Anruf abgebrochen) } }); } class IncomingCall { final String callId; final String roomId; final String callerId; final String? senderName; const IncomingCall({ required this.callId, required this.roomId, required this.callerId, this.senderName, }); } ``` --- ## Call-State-Management (Riverpod) ```dart // lib/features/call/call_provider.dart enum CallState { idle, calling, ringing, connecting, connected, ended, failed } enum CallType { audio, video } @immutable class CallStateData { final CallState state; final CallType type; final String? roomId; final String? callId; final bool isMuted; final bool isCameraOff; final bool isSpeakerOn; final bool isScreenSharing; final List participants; const CallStateData({ this.state = CallState.idle, this.type = CallType.audio, this.roomId, this.callId, this.isMuted = false, this.isCameraOff = false, this.isSpeakerOn = false, this.isScreenSharing = false, this.participants = const [], }); CallStateData copyWith({ CallState? state, CallType? type, String? roomId, String? callId, bool? isMuted, bool? isCameraOff, bool? isSpeakerOn, bool? isScreenSharing, List? participants, }) => CallStateData( state: state ?? this.state, type: type ?? this.type, roomId: roomId ?? this.roomId, callId: callId ?? this.callId, isMuted: isMuted ?? this.isMuted, isCameraOff: isCameraOff ?? this.isCameraOff, isSpeakerOn: isSpeakerOn ?? this.isSpeakerOn, isScreenSharing: isScreenSharing ?? this.isScreenSharing, participants: participants ?? this.participants, ); } class CallNotifier extends StateNotifier { final Ref _ref; LiveKitRoomManager? _manager; MatrixRtcMembership? _membership; CallNotifier(this._ref) : super(const CallStateData()); Future startCall(String roomId, CallType type) async { state = state.copyWith(state: CallState.calling, type: type, roomId: roomId); try { final client = await _ref.read(matrixClientProvider.future); final callId = roomId; const livekitServiceUrl = 'https://livekit.steggi-matrix.work'; _membership = MatrixRtcMembership( roomId: roomId, deviceId: client.deviceID!, callId: callId, livekitServiceUrl: livekitServiceUrl, ); await _membership!.join(client); final credentials = await LiveKitService.getToken( matrixClient: client, roomId: roomId, livekitServiceUrl: livekitServiceUrl, ); _manager = LiveKitRoomManager(); state = state.copyWith(state: CallState.connecting, callId: callId); await _manager!.connect( url: credentials.url, token: credentials.jwt, audioEnabled: true, videoEnabled: type == CallType.video, ); _manager!.onParticipantConnected((p) { state = state.copyWith(participants: [...state.participants, p]); }); _manager!.onParticipantDisconnected((p) { state = state.copyWith( participants: state.participants.where((x) => x.sid != p.sid).toList(), ); }); WakelockPlus.enable(); state = state.copyWith(state: CallState.connected); } catch (e) { await _cleanup(); state = state.copyWith(state: CallState.failed); } } Future hangUp() async { final client = await _ref.read(matrixClientProvider.future); await _membership?.leave(client); await _cleanup(); state = const CallStateData(state: CallState.ended); await Future.delayed(const Duration(seconds: 1)); state = const CallStateData(); } Future toggleMute() async { final muted = !state.isMuted; await _manager?.room.localParticipant?.setMicrophoneEnabled(!muted); state = state.copyWith(isMuted: muted); } Future toggleCamera() async { final off = !state.isCameraOff; await _manager?.room.localParticipant?.setCameraEnabled(!off); state = state.copyWith(isCameraOff: off); } Future toggleSpeaker() async { final on = !state.isSpeakerOn; await Helper.setSpeakerphoneOn(on); state = state.copyWith(isSpeakerOn: on); } Future toggleScreenShare() async { final sharing = !state.isScreenSharing; if (_manager != null) { await _manager!.room.localParticipant?.setScreenShareEnabled(sharing); } state = state.copyWith(isScreenSharing: sharing); } Future _cleanup() async { await _manager?.disconnect(); _manager = null; WakelockPlus.disable(); } } final callProvider = StateNotifierProvider( (ref) => CallNotifier(ref), ); ``` --- ## Call-UI Komponenten ```dart // lib/features/call/call_page.dart class CallPage extends ConsumerWidget { const CallPage({super.key}); @override Widget build(BuildContext context, WidgetRef ref) { final call = ref.watch(callProvider); final notifier = ref.read(callProvider.notifier); return Scaffold( backgroundColor: Colors.black, body: SafeArea( child: Stack( children: [ // ── Haupt-Video (Remote Participant) ────────────────────── if (call.participants.isNotEmpty) Positioned.fill( child: RemoteVideoView(participant: call.participants.first), ) else const Center( child: Icon(Icons.person, size: 120, color: Colors.white24), ), // ── Lokale Kamera (klein, Overlay) ──────────────────────── if (call.type == CallType.video) Positioned( right: 16, top: 16, width: 120, height: 180, child: ClipRRect( borderRadius: BorderRadius.circular(12), child: _LocalPreview(), ), ), // ── Status-Text ─────────────────────────────────────────── Positioned( top: 24, left: 0, right: 0, child: Text( _statusText(call.state), textAlign: TextAlign.center, style: const TextStyle(color: Colors.white, fontSize: 18), ), ), // ── Call-Controls ───────────────────────────────────────── Positioned( bottom: 40, left: 0, right: 0, child: Row( mainAxisAlignment: MainAxisAlignment.spaceEvenly, children: [ _CallButton( icon: call.isMuted ? Icons.mic_off : Icons.mic, label: call.isMuted ? 'Ton an' : 'Stumm', color: call.isMuted ? Colors.red : Colors.white24, onTap: notifier.toggleMute, ), _CallButton( icon: Icons.call_end, label: 'Auflegen', color: Colors.red, size: 72, onTap: notifier.hangUp, ), if (call.type == CallType.video) _CallButton( icon: call.isCameraOff ? Icons.videocam_off : Icons.videocam, label: 'Kamera', color: call.isCameraOff ? Colors.red : Colors.white24, onTap: notifier.toggleCamera, ) else _CallButton( icon: call.isSpeakerOn ? Icons.volume_up : Icons.volume_down, label: 'Lautsprecher', color: Colors.white24, onTap: notifier.toggleSpeaker, ), ], ), ), ], ), ), ); } String _statusText(CallState state) => switch (state) { CallState.calling => 'Verbinde…', CallState.connecting => 'Verbinde…', CallState.connected => 'Verbunden', CallState.ended => 'Aufgelegt', CallState.failed => 'Fehler', _ => '', }; } class _CallButton extends StatelessWidget { final IconData icon; final String label; final Color color; final double size; final VoidCallback onTap; const _CallButton({ required this.icon, required this.label, required this.color, required this.onTap, this.size = 56, }); @override Widget build(BuildContext context) => Column( mainAxisSize: MainAxisSize.min, children: [ GestureDetector( onTap: onTap, child: CircleAvatar( radius: size / 2, backgroundColor: color, child: Icon(icon, color: Colors.white, size: size * 0.5), ), ), const SizedBox(height: 6), Text(label, style: const TextStyle(color: Colors.white70, fontSize: 12)), ], ); } ``` --- ## Hintergrund-Service (Android) ```kotlin // android/app/src/main/kotlin/chat/pyramid/pyramid/CallForegroundService.kt package chat.pyramid.pyramid import android.app.Notification import android.app.NotificationChannel import android.app.NotificationManager import android.app.Service import android.content.Intent import android.os.Build import android.os.IBinder import androidx.core.app.NotificationCompat class CallForegroundService : Service() { override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int { when (intent?.action) { ACTION_START -> startCallNotification() ACTION_STOP -> stopSelf() } return START_STICKY } private fun startCallNotification() { ensureChannel() val notification = NotificationCompat.Builder(this, CHANNEL_ID) .setSmallIcon(R.drawable.ic_notification) .setContentTitle("Pyramid — Aktiver Anruf") .setContentText("Tippe um zum Anruf zurückzukehren") .setOngoing(true) .setPriority(NotificationCompat.PRIORITY_HIGH) .build() if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) { startForeground(NOTIF_ID, notification, android.content.pm.ServiceInfo.FOREGROUND_SERVICE_TYPE_MICROPHONE) } else { startForeground(NOTIF_ID, notification) } } private fun ensureChannel() { if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) { val channel = NotificationChannel( CHANNEL_ID, "Anrufe", NotificationManager.IMPORTANCE_HIGH) (getSystemService(NOTIFICATION_SERVICE) as NotificationManager) .createNotificationChannel(channel) } } override fun onBind(intent: Intent?): IBinder? = null companion object { const val ACTION_START = "START_CALL" const val ACTION_STOP = "STOP_CALL" const val CHANNEL_ID = "pyramid_calls" const val NOTIF_ID = 1001 } } ``` ```dart // Dart: Service starten/stoppen import 'package:flutter/services.dart'; const _callChannel = MethodChannel('chat.pyramid.pyramid/call'); Future startCallService() async { if (!Platform.isAndroid) return; await _callChannel.invokeMethod('startCallService'); } Future stopCallService() async { if (!Platform.isAndroid) return; await _callChannel.invokeMethod('stopCallService'); } ``` --- ## Troubleshooting & Diagnose ### Server-Tests ```bash # LiveKit-Server erreichbar? curl https://livekit.steggi-matrix.work/ # lk-jwt-service erreichbar? curl -X POST https://livekit.steggi-matrix.work/lk-jwt/sfu/get \ -H 'Content-Type: application/json' \ -d '{"room":"!test:server","openid_token":{"access_token":"test","token_type":"Bearer","matrix_server_name":"steggi-matrix.work"},"device_id":"TEST"}' # Erwartete Antwort: {"url":"wss://...","jwt":"eyJ..."} # TURN-Server Test (stuncheck) pip install stun-client python -c "import stun; print(stun.get_ip_info())" # LiveKit Logs sudo journalctl -u livekit -f # lk-jwt-service Logs sudo journalctl -u lk-jwt-service -f # UDP-Port-Reichbarkeit (von außen) nmap -sU -p 50100-50110 ``` ### Häufige Probleme | Problem | Ursache | Lösung | |---------|---------|--------| | `ICE failed` | UDP-Ports gesperrt | TURN aktivieren, Firewall prüfen | | Token-Fehler 401 | OpenID-Validierung schlägt fehl | Matrix-Server-URL in lk-jwt-service prüfen | | `WebSocket closed` | Nginx-Proxy-Timeout | `proxy_read_timeout 3600s` in Nginx | | Kein Audio | Mikrofon-Berechtigung | `Permission.microphone.request()` | | `Room not found` | LiveKit-Key falsch | API-Key in livekit.yaml = lk-jwt-service Config | | Kein Video | Kamera gesperrt | `Permission.camera.request()` | | Call fällt nach 4h ab | `expires_ts` zu niedrig | Bei 30min vor Ablauf Membership erneuern | ### Dart-Diagnose-Tool ```dart Future> diagnoseLiveKit(Client client, String roomId) async { final results = {}; // 1. OpenID-Token try { await client.requestOpenIdToken(client.userID!); results['openid_token'] = true; } catch (_) { results['openid_token'] = false; } // 2. JWT-Service erreichbar try { final r = await http.get( Uri.parse('https://livekit.steggi-matrix.work/')); results['livekit_reachable'] = r.statusCode < 500; } catch (_) { results['livekit_reachable'] = false; } // 3. Token-Austausch try { await LiveKitService.getToken( matrixClient: client, roomId: roomId, livekitServiceUrl: 'https://livekit.steggi-matrix.work', ); results['token_exchange'] = true; } catch (_) { results['token_exchange'] = false; } return results; } ```