GUI Runtime

This page describes the current GUI implemented in src/s6/app/_gui.py and how it is wired up by s6.app.track in UI mode.

Overview

MainWindow is the only GUI entry point in the current codebase. It is used by s6 track --ui, where the track command starts a spawned worker process running s6.app._track_runtime.TrackRuntime. The worker owns playback state, inference, record/snapshot handling, and queue ordering. The Qt process renders the newest processed frame message it receives.

The window is built around four pieces:

  • a fixed-width left side panel showing the current rendered context as YAML;

  • a Vispy SceneCanvas in the center;

  • an optional right-hand plot panel when --enable-plots is set.

  • a bottom transport bar for replay and capture controls.

The canvas itself does not define a fixed layout of views. Instead, the active pipeline populates it through pipeline._setup_views(self) and updates the bound visuals through VisualTree.

Layout

The left side panel shows the current post-pipeline frame context received by the GUI as read-only YAML. The view uses a display-only serializer that converts Pydantic models to plain mappings, summarizes NumPy array leaves as image((shape, ...)), and summarizes long string leaves such as base64 preview payloads as string(<N> chars). When present, the debug mapping is emitted first so pipeline diagnostics stay immediately visible. Basic syntax highlighting separates YAML keys, separators, and scalar values, and refreshes preserve the current scroll position. The active controls live in the bottom transport bar:

  • a play/pause transport action plus one-shot STOP, BACKWARD, and FORWARD actions;

  • a replay-only dataset folder picker;

  • snapshot and record actions;

  • a frame counter;

  • the dataset replay seek slider;

  • a worker-measured pipeline throughput and latency indicator.

The dataset folder picker opens a native directory chooser rooted at ./temp when that directory exists, falling back to the current working directory. It is enabled only for dataset replay workers. EV offset and contrast sliders are hidden for now; the GUI no longer emits EV/contrast commands.

Replay mode enables the frame seek slider once the worker reports a positive frame_length. Live mode keeps the dataset picker, seek slider, and replay buttons disabled. Dragging or clicking the replay slider sends GOTOFRAME requests for the selected frame positions. The worker pauses replay, runs inference for the most recent requested frame, and emits that processed result.

Runtime Loop

MainWindow.show_canvas() stores the worker output queue, shows the canvas, and starts a Vispy timer with a 0.005 second interval. Each timer tick performs a non-blocking queue drain and renders only the newest available TrackFrameMessage:

  • the newest message context is passed to VisualTree.update(context);

  • the canvas is repainted;

  • the seek bar, frame counter, playback action, and record toggle are synchronized from the accompanying TrackState;

  • the pipeline stats label is updated from the worker-measured TrackState.pipeline_fps and TrackState.pipeline_latency_ms values;

  • the left sidebar is refreshed from the rendered context;

  • when plots are enabled, solver.tippoint_b is appended to the X/Y/Z traces.

The pipeline stats indicator measures only the worker’s pipeline call, excluding dataset read time, replay pacing sleeps, queue publishing, and dataset recording writes. It shows Pipeline: -- FPS | -- ms until the worker reports values, then displays throughput and average per-frame inference latency.

The GUI no longer infers replay state from frame numbers. During an in-flight slider seek, it keeps the requested frame counter/slider position and suppresses replay messages until the worker acknowledges the requested frame in a paused state. This protects the UI from already-queued pre-seek playback messages. The worker still drops stale queue items before publishing each new message.

The plot panel shows tip-point position in micrometers versus time, keeping the most recent 100 samples. It uses context["timestamp"] when available and falls back to sample count if the timestamp is missing.

Commands Emitted

The GUI sends commands through command_queue.put((CommandSet, payload)).

  • TAKESNAPSHOT with None from the snapshot action.

  • RECORD with True or False from the record toggle.

  • CHANGEDATASET with the selected dataset directory from the replay-only Load Dataset action.

  • PLAYBACKCONTROL with a PlaybackMode value from the transport buttons.

  • GOTOFRAME with an integer frame number from the seek bar as the slider moves.

Seeking leaves replay paused because the worker treats GOTOFRAME as an atomic pause-and-render operation. Clicking the seek bar jumps directly to the clicked frame instead of using Qt’s page-step behavior. BACKWARD and FORWARD also pause replay, render the adjacent frame, and leave playback paused. The play/pause action shows Pause while replay is playing and Play while replay is paused or stopped. Stop is a one-shot action and is never displayed as selected.

In UI mode, s6 track -o ... starts dataset recording immediately and the record action shows the active recording state once the worker publishes its first state message. Without -o, track prepares a generated temp/track_* dataset path and records to it only after the record action is clicked on.

Notes

  • The GUI assumes the caller provides a command_queue; the current runtime does this from s6.app.track in UI mode.

  • The left panel width is fixed at 450 pixels and the plot panel width is fixed at 400 pixels.

  • The visual bindings are pipeline-specific. The GUI does not hard-code image names or 3D views beyond handing the canvas to the pipeline.