Video

Plinken supports video tracks for scoring to picture, a GPU-accelerated compositor for building visual scenes, live camera input, ISF shader effects, and multi-protocol live streaming.

Architecture

Video file / Camera / Generator
        ↓
  VideoDecoder (per-backend)
        ↓
  Preview Cache (720p NV12 ring buffer)
        ↓
  GPU Pipeline (wgpu)
        ↓
  DAG Renderer → Output (IOSurface / CVPixelBuffer)

Decoding Backends

Platform	Backend	Notes
macOS	AVFoundation	Native, hardware-accelerated, zero-copy
Cross-platform	FFmpeg	Requires `ffmpeg` feature flag
Windows	Media Foundation	Planned
Linux	GStreamer	Planned

Video Module

The Video module (video) manages playback state. The decoder and frame cache are owned by the Video Panel for thread safety.

Module Parameters

Parameter	Type	Description
`visible`	bool	Video panel visibility
`fullscreen`	bool	Fullscreen playback mode
`studio_mode`	bool	Studio Mode — dual preview/program view
`broadcasting`	bool	Whether the broadcast stream is active
`video_path`	string	Path to the currently loaded video file
`current_frame`	int	Current frame number (read-only, derived from transport)

Video Metadata (read-only)

Once a video is loaded, the module exposes:

Property	Type	Description
Dimensions	(u32, u32)	Width × height in pixels
Frame rate	f64	Frames per second
Total frames	u64	Total frame count
Duration	f64	Duration in seconds

Commands

Command ID	Name	Description
`video:toggle-visible`	Show Video	Toggle the video panel
`video:toggle-fullscreen`	Toggle Fullscreen	Enter/exit fullscreen video playback
`video:toggle-broadcast`	Toggle Broadcast	Start/stop live streaming output
`video:load`	Load Video	Open a video file for playback
`video:frame-forward`	Frame Forward	Step one frame forward
`video:frame-back`	Frame Back	Step one frame back
`video:capture-frame`	Capture Frame	Save current frame as PNG
`video:extract-mask`	Extract Mask	Run Vision segmentation on current frame

Lua API

-- Toggle video panel
pl.cmd("video:toggle-visible")

-- Load a video file
pl.cmd("video:load", { path = "/path/to/video.mp4" })

-- Step through frames
pl.cmd("video:frame-forward")
pl.cmd("video:frame-back")

-- Capture the current frame as PNG
pl.cmd("video:capture-frame", {
  output_path = "/tmp/frame.png",
  mode = "final"  -- "preview" or "final"
})

-- Start broadcasting
pl.cmd("video:toggle-broadcast")

-- Read video state
local visible = plinken.get("video", "visible")
local frame   = plinken.get("video", "current_frame")

Frame-Accurate Sync

Video playback is synced to the audio transport. The current frame is calculated from the audio sample position:

frame = floor(sample_position / sample_rate × frame_rate)

This ensures frame-accurate lip-sync and hit-point alignment when scoring to picture.

Preview Cache

The preview cache stores decoded frames at 720p NV12 in a ring buffer for low-latency playback without keyframe seeking. The background decoder pre-decodes frames ahead of the playhead for smooth scrubbing.

Setting	Value
Preview resolution	1280 × 720
Format	NV12 (YCbCr 4:2:0)
Strategy	Ring buffer with background decode

Frame Capture

Capture the current compositor frame as a PNG image. Two quality modes:

Mode	Description
`preview`	Screen-resolution capture (fast, used during playback)
`final`	Full-resolution render (slow, used when stopped)

Vision Segmentation

The Extract Mask command uses Apple Vision framework to detect subjects in the current frame. Detected subjects can be used to create mask layers for compositing.

Keyboard Shortcuts

Shortcut	Action
`←` / `→`	Frame back / forward (when video panel focused)
`Cmd+L`	Go Live (start streaming)
`Shift+L`	Toggle Studio Mode

Compositor — Layer system, blend modes, transforms
Volume FX — 3D volumetric effects
VFX Presets — Preset library
Live Streaming — WHEP, RTMP, SRT protocols