GUI Architecture & Interaction¶

This document details the GUI subsystem implemented in _gui.py.
It covers layout, widgets, data flow, commands emitted to the backend, and the update loop.

1. High-Level Layout¶

The main window (MainWindow) arranges three vertical zones:

+------------------------------ MainWindow ------------------------------+
| [Left: Control Panel ~250px] |         [Center: Vispy Canvas]         | [Right: Plot ~400px]
|                              |  Grid of image views + 3D (optional)    | Tip-point X/Y/Z traces
|  - Capture Toolbar           |  - L/R/B images, warp, render, tracking | (pyqtgraph)
|  - Dataset Selector          |  - VisualTree binds & updates           |
|  - EV / Contrast Controls    |                                         |
|  - Robotic Control Panel     |                                         |
|  - Logger (QPlainTextEdit)   |                                         |
+-----------------------------------------------------------------------+

  • Control Panel (left, fixed width ~250): capture/record, dataset switch, exposure/contrast, robotic control, and a live log console.

  • Canvas (center): a Vispy SceneCanvas hosting a grid of views. Each view shows an image (e.g., L.image, R.image, B.render, etc.) or 3D visuals (frusta, meshes).

  • Plot Panel (right, fixed width ~400): three pyqtgraph plots (X, Y, Z vs time) visualizing solver tip-point traces.

2. Modes & Initialization¶

MainWindow(replay: bool, enable_plots: bool=False, command_queue: Queue=None, dataset_path: str="None"):

  • Replay mode (replay=True)

    • Shows PlaybackControlWidget (transport + seek bar).

    • Hides live-only controls (e.g., snapshot/record toolbar and robotic panel).

  • Live mode (replay=False)

    • Shows CaptureToolbar (snapshot/record with overlay/flash).

    • Shows RoboticControlPanel.

  • Common

    • DatasetSelectorWidget to set dataset folder or show Network if none.

    • EvControlWidget / ContrastControlWidget for exposure/contrast tuning.

    • Log terminal wired via a QTextEditLogger handler.

3. Canvas & Visual Binding¶

self.visual_tree = VisualTree()
self.canvas = scene.SceneCanvas(keys="interactive", size=(1920, 1080))
self.grid   = self.canvas.central_widget.add_grid()
self.cameras = load_cameras()  # used if enabling 3D viewport bindings

3.1 Views¶

The grid places multiple PanZoomCamera image views (exact rows/cols configured in _setup_views()):

  • Left/Right (L/R): raw BGR and processed images

    • bindings like ["L.bgr_image", "L.image"], ["R.bgr_image", "R.image"]

  • Back (B): render and tracking views

    • bindings like "B.render", "B.tracking_image"

  • Warp previews: "L.warp.image", "R.warp.image"

Each image view is created via image(view, binding, resolution), which:

  • sets view.bgcolor = "#150000"

  • binds an Image visual to binding through VisualTree.bind(...)

  • uses scene.PanZoomCamera(aspect=1) and sets range to the provided resolution

Note: A 3D viewport (viewport(view)) is wired with frustum and mesh visuals but is currently commented out by default. It binds:

  • Frustum3D for L/R warp cameras and loaded cameras.

  • ExternalMeshVisual ("assets/CatAR v2 Assembly v23.obj") and a Mesh3D("assets/test.obj") aligned to solver markers.

  • Marker3D for "solver.tippoint".

4. Update Loop & Context Contract¶

A Vispy app.Timer(interval=0.005) drives _update(event):

if self.q is not None and not self.q.empty():
    context = self.q.get()
    self.visual_tree.update(context)   # pushes data into bound visuals
    self.canvas.update()

  • When first context arrives, the total frame count (context["frame_length"]) initializes the replay seek bar.

  • The current frame index (context["frame_serial"]) is used to keep UI in sync.

  • If solver.tippoint_b is present, tip traces are recorded and plotted (micrometers vs time):

    • X → red, Y → green, Z → blue

    • time base uses context["timestamp"] (fallback to sample count)

Expected context keys (partial, inferred from bindings):

  • frame_length: int, frame_serial: int, timestamp: float

  • L.image, L.bgr_image, R.image, R.bgr_image

  • L.warp.image, R.warp.image, L.warp.camera, R.warp.camera

  • B.render, B.tracking_image

  • solver.tippoint / solver.tippoint_b, solver.instrument_1, solver.marker_points.*

The exact schema is governed by pipeline/stage outputs; the GUI consumes whatever is present and bound.

5. Control Panel Widgets¶

5.1 PlaybackControlWidget (Replay mode)¶

  • Toolbar with actions: prev / play / pause / stop / next

  • Seek slider (pageStep=30) and index label "{k+1} / {total}".

  • Emits to the backend via command_queue:

    • PLAYBACKCONTROL, PlaybackMode.PLAY|PAUSE|STOP|BACKWARD|FORWARD

    • GOTOFRAME, k (when seek released)

  • Debounced scrubbing: UI updates label during drag; command sent on release.

5.2 CaptureToolbar (Live mode)¶

  • Buttons: Take Snapshot and Start/Stop Recording (native icons).

  • On snapshot:

    • UI flash via overlay effect

    • TAKESNAPSHOT command sent

  • On record toggle:

    • Starts a blinking border overlay around the target widget (canvas.native by default)

    • RECORD, True|False command sent

  • Overlay implementation uses:

    • QGraphicsOpacityEffect + QPropertyAnimation for flash/blink

    • An always-on-top child overlay (QFrame) following the target’s geometry (updated on Resize / Move via eventFilter)

5.3 DatasetSelectorWidget¶

  • Shows current dataset path (read-only line edit) and a folder button.

  • On folder pick → updates path and emits: CHANGEDATASET, "<dir>".

  • Displays "Network" when no dataset path was supplied.

5.4 Exposure / Contrast Controls¶

  • EvControlWidget:

    • Range: [-5.0, +5.0], step 1/3 EV

    • valueChanged updates label and emits SETEVOFFSET, float(ev)

  • ContrastControlWidget:

    • EV-like contrast k in [-2.0, +2.0], step 0.1

    • valueChanged updates label and emits:

      • SETCONTRAST, float(k) (continuous)

      • SET_CONTRAST_EV, float(k) (on change with optional notify)

5.5 RoboticControlPanel (Live mode)¶

  • Embedded from s6.ui.robotic_control_panel.RoboticControlPanel.

  • Receives and dispatches robotic related actions (details in that module).

5.6 Logger¶

  • QPlainTextEdit (read-only) wired via QTextEditLogger(logging.Handler) to display runtime logs.

6. Commands Emitted by the GUI¶

All commands are sent as tuples via command_queue.put((CommandSet.<CMD>, payload)).

GUI Action

CommandSet

Payload

Take snapshot

TAKESNAPSHOT

None

Start/stop recording

RECORD

True / False

Change dataset dir

CHANGEDATASET

str path

Set EV offset

SETEVOFFSET

float

Set contrast (continuous)

SETCONTRAST

float

Set contrast (EV-style step)

SET_CONTRAST_EV

float

Playback: play

PLAYBACKCONTROL

PlaybackMode.PLAY

Playback: pause

PLAYBACKCONTROL

PlaybackMode.PAUSE

Playback: stop

PLAYBACKCONTROL

PlaybackMode.STOP

Playback: prev (reverse)

PLAYBACKCONTROL

PlaybackMode.BACKWARD

Playback: next (forward)

PLAYBACKCONTROL

PlaybackMode.FORWARD

Seek to frame k

GOTOFRAME

int k

PlaybackMode values: PLAY, PAUSE, STOP, BACKWARD, FORWARD.

7. Data Flow¶

7.1 Event Flow (User → Backend)¶

        sequenceDiagram
    participant UI as GUI Widgets
    participant CQ as command_queue (to backend)
    participant BE as Backend (context generator + pipeline)

    UI->>CQ: (CommandSet, payload)
    Note right of CQ: Transport: multiprocessing.Queue
    CQ-->>BE: Dequeue & handle command
    BE-->>BE: Update source/pipeline state (e.g., play/pause, seek, record, EV, contrast)

7.2 Telemetry Flow (Backend → GUI)¶

        sequenceDiagram
    participant BE as Backend (producer)
    participant Q as Queue (context updates)
    participant MW as MainWindow._update()

    BE->>Q: context dict (images, solver, metadata)
    MW->>Q: non-blocking poll (q.empty? get())
    Q-->>MW: next context
    MW->>VisualTree: update(context)
    MW->>Canvas: request repaint
    MW->>Plots: append tip-point traces

8. Timing & Performance Notes¶

  • Timer interval: ~5ms (app.Timer(interval=0.005)), subject to main-thread GUI constraints.

  • q.empty() is used to avoid blocking; bursty producers may fill the queue—consider backpressure if needed.

  • Visuals update is incremental via VisualTree.update(context).

  • Plots keep a deque of 100 samples per axis; adjust if longer histories are desired.

9. Extensibility Guidelines¶

  • To add a new per-frame visual:

    1. Add a new view cell via _setup_views() and call image(view, binding, resolution) (or bind a 3D visual in viewport(view)).

    2. Ensure the backend publishes the matching context key used in binding.

  • To add a new control:

    1. Implement a small QWidget with signals/slots.

    2. Map UI events to (CommandSet, payload) on command_queue.

    3. Handle that command in the backend.

10. Known Assumptions¶

  • The context schema is produced by the pipeline stages; GUI is tolerant of missing keys but the plotted tip-point requires solver.tippoint_b.

  • 3D viewport is optional; it binds camera frusta and meshes if enabled.

  • Dataset path selection is only meaningful in replay/offline workflows.