---
title: GUI Architecture & Interaction
last_updated: 2025-11-05 MingHan
---

# GUI Architecture & Interaction

This document details the GUI subsystem implemented in `_gui.py`.  
It covers layout, widgets, data flow, commands emitted to the backend, and the update loop.

---

## 1. High-Level Layout

The main window (`MainWindow`) arranges three vertical zones:

```
+------------------------------ MainWindow ------------------------------+
| [Left: Control Panel ~250px] |         [Center: Vispy Canvas]         | [Right: Plot ~400px]
|                              |  Grid of image views + 3D (optional)    | Tip-point X/Y/Z traces
|  - Capture Toolbar           |  - L/R/B images, warp, render, tracking | (pyqtgraph)
|  - Dataset Selector          |  - VisualTree binds & updates           |
|  - EV / Contrast Controls    |                                         |
|  - Robotic Control Panel     |                                         |
|  - Logger (QPlainTextEdit)   |                                         |
+-----------------------------------------------------------------------+
```

- **Control Panel (left, fixed width ~250)**: capture/record, dataset switch, exposure/contrast, robotic control, and a live log console.
- **Canvas (center)**: a Vispy `SceneCanvas` hosting a grid of views. Each view shows an image (e.g., `L.image`, `R.image`, `B.render`, etc.) or 3D visuals (frusta, meshes).
- **Plot Panel (right, fixed width ~400)**: three pyqtgraph plots (X, Y, Z vs time) visualizing solver tip-point traces.

---

## 2. Modes & Initialization

`MainWindow(replay: bool, enable_plots: bool=False, command_queue: Queue=None, dataset_path: str="None")`:

- **Replay mode (`replay=True`)**
  - Shows **PlaybackControlWidget** (transport + seek bar).
  - Hides live-only controls (e.g., snapshot/record toolbar and robotic panel).
- **Live mode (`replay=False`)**
  - Shows **CaptureToolbar** (snapshot/record with overlay/flash).
  - Shows **RoboticControlPanel**.
- **Common**
  - **DatasetSelectorWidget** to set dataset folder or show `Network` if none.
  - **EvControlWidget** / **ContrastControlWidget** for exposure/contrast tuning.
  - **Log terminal** wired via a `QTextEditLogger` handler.

---

## 3. Canvas & Visual Binding

```python
self.visual_tree = VisualTree()
self.canvas = scene.SceneCanvas(keys="interactive", size=(1920, 1080))
self.grid   = self.canvas.central_widget.add_grid()
self.cameras = load_cameras()  # used if enabling 3D viewport bindings
```

### 3.1 Views

The grid places multiple `PanZoomCamera` image views (exact rows/cols configured in `_setup_views()`):

- **Left/Right (L/R)**: raw BGR and processed images  
  - bindings like `["L.bgr_image", "L.image"]`, `["R.bgr_image", "R.image"]`
- **Back (B)**: render and tracking views  
  - bindings like `"B.render"`, `"B.tracking_image"`
- **Warp previews**: `"L.warp.image"`, `"R.warp.image"`

Each image view is created via `image(view, binding, resolution)`, which:
- sets `view.bgcolor = "#150000"`
- binds an `Image` visual to `binding` through `VisualTree.bind(...)`
- uses `scene.PanZoomCamera(aspect=1)` and sets range to the provided `resolution`

> Note: A 3D viewport (`viewport(view)`) is wired with frustum and mesh visuals but is currently commented out by default. It binds:
> - `Frustum3D` for L/R warp cameras and loaded cameras.
> - `ExternalMeshVisual` (`"assets/CatAR v2 Assembly v23.obj"`) and a `Mesh3D("assets/test.obj")` aligned to solver markers.
> - `Marker3D` for `"solver.tippoint"`.

---

## 4. Update Loop & Context Contract

A Vispy `app.Timer(interval=0.005)` drives `_update(event)`:

```python
if self.q is not None and not self.q.empty():
    context = self.q.get()
    self.visual_tree.update(context)   # pushes data into bound visuals
    self.canvas.update()
```

- When first context arrives, the total frame count (`context["frame_length"]`) initializes the replay seek bar.
- The current frame index (`context["frame_serial"]`) is used to keep UI in sync.
- If `solver.tippoint_b` is present, tip traces are recorded and plotted (micrometers vs time):
  - X → red, Y → green, Z → blue
  - time base uses `context["timestamp"]` (fallback to sample count)

**Expected context keys (partial, inferred from bindings):**
- `frame_length: int`, `frame_serial: int`, `timestamp: float`
- `L.image`, `L.bgr_image`, `R.image`, `R.bgr_image`
- `L.warp.image`, `R.warp.image`, `L.warp.camera`, `R.warp.camera`
- `B.render`, `B.tracking_image`
- `solver.tippoint` / `solver.tippoint_b`, `solver.instrument_1`, `solver.marker_points.*`

> The exact schema is governed by pipeline/stage outputs; the GUI consumes whatever is present and bound.

---

## 5. Control Panel Widgets

### 5.1 PlaybackControlWidget (Replay mode)
- Toolbar with actions: **prev / play / pause / stop / next**
- **Seek slider** (pageStep=30) and **index label** `"{k+1} / {total}"`.
- Emits to the backend via `command_queue`:
  - `PLAYBACKCONTROL, PlaybackMode.PLAY|PAUSE|STOP|BACKWARD|FORWARD`
  - `GOTOFRAME, k` (when seek released)
- Debounced scrubbing: UI updates label during drag; command sent on release.

### 5.2 CaptureToolbar (Live mode)
- Buttons: **Take Snapshot** and **Start/Stop Recording** (native icons).
- On snapshot:
  - UI flash via overlay effect
  - `TAKESNAPSHOT` command sent
- On record toggle:
  - Starts a blinking border overlay around the target widget (`canvas.native` by default)
  - `RECORD, True|False` command sent
- Overlay implementation uses:
  - `QGraphicsOpacityEffect` + `QPropertyAnimation` for flash/blink
  - An always-on-top child overlay (`QFrame`) following the target’s geometry (updated on `Resize` / `Move` via `eventFilter`)

### 5.3 DatasetSelectorWidget
- Shows current dataset path (read-only line edit) and a folder button.
- On folder pick → updates path and emits: `CHANGEDATASET, "<dir>"`.
- Displays `"Network"` when no dataset path was supplied.

### 5.4 Exposure / Contrast Controls
- **EvControlWidget**:
  - Range: `[-5.0, +5.0]`, step `1/3 EV`
  - `valueChanged` updates label and emits `SETEVOFFSET, float(ev)`
- **ContrastControlWidget**:
  - EV-like contrast `k` in `[-2.0, +2.0]`, step `0.1`
  - `valueChanged` updates label and emits:
    - `SETCONTRAST, float(k)`   (continuous)
    - `SET_CONTRAST_EV, float(k)` (on change with optional notify)

### 5.5 RoboticControlPanel (Live mode)
- Embedded from `s6.ui.robotic_control_panel.RoboticControlPanel`.
- Receives and dispatches robotic related actions (details in that module).

### 5.6 Logger
- `QPlainTextEdit` (read-only) wired via `QTextEditLogger(logging.Handler)` to display runtime logs.

---

## 6. Commands Emitted by the GUI

All commands are sent as tuples via `command_queue.put((CommandSet.<CMD>, payload))`.

| GUI Action | CommandSet | Payload |
|------------|------------|---------|
| Take snapshot | `TAKESNAPSHOT` | `None` |
| Start/stop recording | `RECORD` | `True` / `False` |
| Change dataset dir | `CHANGEDATASET` | `str` path |
| Set EV offset | `SETEVOFFSET` | `float` |
| Set contrast (continuous) | `SETCONTRAST` | `float` |
| Set contrast (EV-style step) | `SET_CONTRAST_EV` | `float` |
| Playback: play | `PLAYBACKCONTROL` | `PlaybackMode.PLAY` |
| Playback: pause | `PLAYBACKCONTROL` | `PlaybackMode.PAUSE` |
| Playback: stop | `PLAYBACKCONTROL` | `PlaybackMode.STOP` |
| Playback: prev (reverse) | `PLAYBACKCONTROL` | `PlaybackMode.BACKWARD` |
| Playback: next (forward) | `PLAYBACKCONTROL` | `PlaybackMode.FORWARD` |
| Seek to frame *k* | `GOTOFRAME` | `int` k |

**PlaybackMode values:** `PLAY`, `PAUSE`, `STOP`, `BACKWARD`, `FORWARD`.

---

## 7. Data Flow

### 7.1 Event Flow (User → Backend)
``` {mermaid}
sequenceDiagram
    participant UI as GUI Widgets
    participant CQ as command_queue (to backend)
    participant BE as Backend (context generator + pipeline)

    UI->>CQ: (CommandSet, payload)
    Note right of CQ: Transport: multiprocessing.Queue
    CQ-->>BE: Dequeue & handle command
    BE-->>BE: Update source/pipeline state (e.g., play/pause, seek, record, EV, contrast)
```

### 7.2 Telemetry Flow (Backend → GUI)
``` {mermaid}
sequenceDiagram
    participant BE as Backend (producer)
    participant Q as Queue (context updates)
    participant MW as MainWindow._update()

    BE->>Q: context dict (images, solver, metadata)
    MW->>Q: non-blocking poll (q.empty? get())
    Q-->>MW: next context
    MW->>VisualTree: update(context)
    MW->>Canvas: request repaint
    MW->>Plots: append tip-point traces
```

---

## 8. Timing & Performance Notes

- **Timer interval**: ~5ms (`app.Timer(interval=0.005)`), subject to main-thread GUI constraints.
- `q.empty()` is used to avoid blocking; bursty producers may fill the queue—consider backpressure if needed.
- Visuals update is incremental via `VisualTree.update(context)`.
- Plots keep a deque of 100 samples per axis; adjust if longer histories are desired.

---

## 9. Extensibility Guidelines

- To add a new per-frame visual:
  1. Add a new view cell via `_setup_views()` and call `image(view, binding, resolution)` (or bind a 3D visual in `viewport(view)`).
  2. Ensure the backend publishes the matching `context` key used in `binding`.
- To add a new control:
  1. Implement a small `QWidget` with signals/slots.
  2. Map UI events to `(CommandSet, payload)` on `command_queue`.
  3. Handle that command in the backend.

---

## 10. Known Assumptions

- The context schema is produced by the pipeline stages; GUI is tolerant of missing keys but the plotted tip-point requires `solver.tippoint_b`.
- 3D viewport is optional; it binds camera frusta and meshes if enabled.
- Dataset path selection is only meaningful in replay/offline workflows.

---