Skip to content

capture

capture #

World-tracked capture procedures backed by ARKit.

Two sampling regimes over the same posed RGB-D substrate, both feeding offline 3D reconstruction (e.g. gaussian splatting on a server):

  • ARKitCaptureScene — a sparse, curated set of posed keyframes, novelty- and sharpness-gated. Maximal data per viewpoint; a keyframe carries everything the tracker can deliver, so derived products (meshes, point clouds, trajectories) are reconstructed server-side rather than captured as lesser substitutes.
  • ARKitCaptureVideo — the dense, continuous counterpart: one clip with the 6-DoF pose baked into every frame and best-effort depth on the (sparser) frames a depth sensor reached.
Example
import capturegraph as cg


class SceneTarget(cg.Struct):
    world_map: cg.ARWorldMap
    sessions: cg.Map[cg.Date, cg.SceneCapture]


@cg.procedure(SceneTarget)
def scene(root: cg.Procedure[cg.Path[SceneTarget]]):
    session = root.sessions[cg.CaptureTime()]
    capture = cg.ARKitCaptureScene(
        initial_world_map=root.world_map.load_or(cg.ARKitCreateScene()),
    )
    # The full capture per session (each doubles as a dated world-map
    # backup); the shared slot carries the current map forward.
    session &= capture
    root.world_map &= capture.world_map

ARKitCaptureScene #

Bases: Procedure[SceneCapture]

Capture a world-anchored set of posed keyframes with ARKit world tracking.

The user walks the scene while keyframes fire automatically whenever the camera has moved or rotated enough past the frames already captured (the translation and rotation fractions of the thresholds combine as a Euclidean norm against the nearest kept keyframe, so partial motion on both axes accumulates) — and is moving slowly enough for a sharp frame (the translational and angular speeds, as fractions of max_speed_mps and max_rotation_dps, combine the same way; while their norm exceeds 1 the shutter is held, so an overspeeding camera cannot fire a motion-blurred keyframe. The speeds are peak-held with a short exponential decay, so a momentary dip mid-swing does not read as steady — captures wait for a genuine settle). Each keyframe reduces from a single tracked frame: the image, its 6-DoF world pose with pinhole intrinsics, its exposure/light estimate, a best-effort metric depth map captured in the same instant (present on devices that deliver scene depth), and a best-effort GPS fix (absent with geotagging disabled or no signal) — pose/fix pairs let a server solve the scene's geographic alignment, and the rest is the raw substrate it reconstructs offline. Finishing also serializes the session's ARWorldMap, so a later session (any participant) relocalizes into the same coordinate frame and extends the scene.

Attributes:

Name Type Description
initial_world_map Input[ARWorldMap]

The scene's world map to relocalize into. Wire it as root.world_map.load_or(cg.ARKitCreateScene()) — the stored map when the scene exists, or a brand-new scene iff nothing is stored yet. (Named for what it is at session start, and so capture.world_map stays free to navigate the result's refreshed map.)

min_translation_meters Input[Number]

Metres of camera movement that alone earn the next keyframe (default: 0.1).

min_rotation_degrees Input[Number]

Degrees of camera rotation that alone earn the next keyframe (default: 10.0).

max_speed_mps Input[Number]

Metres per second of camera movement past which a capture would motion-blur; the shutter is held until the camera slows (default: 0.5 — ~8 px of blur on the 4K stream at a 1/60 s indoor exposure and ~3 m scene depth, which is ~4 px at the roughly halved resolution reconstruction runs at; translational blur scales inversely with depth, so tighten this for close-up scenes).

max_rotation_dps Input[Number]

Degrees per second of camera rotation past which a capture would motion-blur, holding the shutter the same way (default: 10.0 — the same budget, depth-independent: the ~65° wide camera smears ~0.9 px per °/s at 4K and 1/60 s, and OIS recovers much of so small an angular excursion; matches the panorama gate's rate limit).

max_frames Input[Number]

Soft cap on keyframes per session (default: 300).

format Input[PosedVideoFormat]

The frame-rate floor / resolution priority for the AR stream each keyframe is cut from (default: PosedVideoFormat.RESOLUTION) — see PosedVideoFormat for the exact semantics of each case (no slow-motion case: ARKit tracking tops out at 60 Hz).

Keyframes are always 8-bit sRGB JPEG — the reconstruction toolchain (COLMAP/FreeImage, OpenCV) reads JPEG directly, and the fixed sRGB target keeps every frame colorimetrically uniform. The session always prefers an HDR-capable video format (best-effort): ARKit's video HDR is exposure fusion within that same 8-bit feed, so it improves highlight retention without changing the on-disk format.

Example
capture = cg.ARKitCaptureScene(
    initial_world_map=root.world_map.load_or(cg.ARKitCreateScene()),
    label="Capture the scene",
)
session.frames &= capture.frames
root.world_map &= capture.world_map
Source code in capturegraph-lib/capturegraph/procedures/nodes/collection/arkit/capture.py
@make_procedure
class ARKitCaptureScene(Procedure[SceneCapture]):
    """Capture a world-anchored set of posed keyframes with ARKit world tracking.

    The user walks the scene while keyframes fire automatically whenever the
    camera has moved or rotated enough past the frames already captured (the
    translation and rotation fractions of the thresholds combine as a Euclidean
    norm against the nearest kept keyframe, so partial motion on both axes
    accumulates) — and is moving slowly enough for a sharp frame (the
    translational and angular speeds, as fractions of ``max_speed_mps`` and
    ``max_rotation_dps``, combine the same way; while their norm exceeds 1 the
    shutter is held, so an overspeeding camera cannot fire a motion-blurred
    keyframe. The speeds are peak-held with a short exponential decay, so a
    momentary dip mid-swing does not read as steady — captures wait for a
    genuine settle). Each keyframe reduces from a single tracked frame: the
    image,
    its 6-DoF world pose with pinhole intrinsics, its exposure/light estimate,
    a best-effort metric depth map captured in the same instant (present on
    devices that deliver scene depth), and a best-effort GPS fix (absent with
    geotagging disabled or no signal) — pose/fix pairs let a server solve the
    scene's geographic alignment, and the rest is the raw substrate it
    reconstructs offline. Finishing also serializes the session's
    ``ARWorldMap``, so a later session (any participant) relocalizes into the
    same coordinate frame and extends the scene.

    Attributes:
        initial_world_map: The scene's world map to relocalize into. Wire it as
            ``root.world_map.load_or(cg.ARKitCreateScene())`` — the stored map when
            the scene exists, or a brand-new scene iff nothing is stored yet.
            (Named for what it is at session start, and so ``capture.world_map``
            stays free to navigate the *result's* refreshed map.)
        min_translation_meters: Metres of camera movement that alone earn the
            next keyframe (default: 0.1).
        min_rotation_degrees: Degrees of camera rotation that alone earn the
            next keyframe (default: 10.0).
        max_speed_mps: Metres per second of camera movement past which a capture
            would motion-blur; the shutter is held until the camera slows
            (default: 0.5 — ~8 px of blur on the 4K stream at a 1/60 s indoor
            exposure and ~3 m scene depth, which is ~4 px at the roughly
            halved resolution reconstruction runs at; translational blur
            scales inversely with depth, so tighten this for close-up
            scenes).
        max_rotation_dps: Degrees per second of camera rotation past which a
            capture would motion-blur, holding the shutter the same way
            (default: 10.0 — the same budget, depth-independent: the ~65°
            wide camera smears ~0.9 px per °/s at 4K and 1/60 s, and OIS
            recovers much of so small an angular excursion; matches the
            panorama gate's rate limit).
        max_frames: Soft cap on keyframes per session (default: 300).
        format: The frame-rate floor / resolution priority for the AR stream
            each keyframe is cut from (default:
            ``PosedVideoFormat.RESOLUTION``) — see ``PosedVideoFormat``
            for the exact semantics of each case (no slow-motion case: ARKit
            tracking tops out at 60 Hz).

    Keyframes are always 8-bit sRGB JPEG — the reconstruction toolchain
    (COLMAP/FreeImage, OpenCV) reads JPEG directly, and the fixed sRGB target
    keeps every frame colorimetrically uniform. The session always prefers an
    HDR-capable video format (best-effort): ARKit's video HDR is exposure
    fusion within that same 8-bit feed, so it improves highlight retention
    without changing the on-disk format.

    Example:
        ```python
        capture = cg.ARKitCaptureScene(
            initial_world_map=root.world_map.load_or(cg.ARKitCreateScene()),
            label="Capture the scene",
        )
        session.frames &= capture.frames
        root.world_map &= capture.world_map
        ```
    """

    initial_world_map: Input[ARWorldMap]
    min_translation_meters: Input[Number] = 0.10
    min_rotation_degrees: Input[Number] = 10.0
    max_speed_mps: Input[Number] = 0.5
    max_rotation_dps: Input[Number] = 10.0
    max_frames: Input[Number] = 300
    format: Input[PosedVideoFormat] = PosedVideoFormat.RESOLUTION

ARKitCaptureVideo #

Bases: Procedure[PosedVideo]

Record a world-anchored posed RGB-D video with ARKit world tracking.

The dense, continuous counterpart to ARKitCaptureScene. Where a scene scan keeps a sparse, sharpness-gated set of posed keyframes, this records one continuous clip and bakes the 6-DoF camera pose into every frame, plus a best-effort metric depth map on the frames a depth sensor reaches (LiDAR/depth-camera devices). Depth is sparser than the RGB stream, so it is kept keyed by the exact frame it was measured at — the result is honest about which frames carry fresh depth rather than holding a stale map forward. Finishing archives the session's ARWorldMap the same way ARKitCaptureScene does, so the video shares a scene's coordinate frame and later sessions relocalize into it and extend the capture.

The result is a PosedVideo: the clip, its per-frame poses and depth (both keyed by frame instant so they line up), and the refreshed world_map. Reach into the result's fields to store or forward them — capture.video, capture.poses, capture.depth, capture.world_map.

Attributes:

Name Type Description
initial_world_map Input[ARWorldMap]

The scene's world map to relocalize into. Wire it as root.world_map.load_or(cg.ARKitCreateScene()) — the stored map when the scene exists, or a brand-new scene iff nothing is stored yet. (Named for what it is at session start, so capture.world_map stays free to navigate the result's refreshed map.)

dynamic_range Input[DynamicRange]

Whether the session engages ARKit's video HDR — exposure fusion within the unchanged 8-bit BT.709 feed, better highlight retention — when the AR video format offers it (default: DynamicRange.SDR); best-effort, see DynamicRange.

format Input[PosedVideoFormat]

The frame-rate floor / resolution priority (default: PosedVideoFormat.RESOLUTION) — see PosedVideoFormat for the exact semantics of each case (no slow-motion case: ARKit tracking tops out at 60 Hz).

Example
capture = cg.ARKitCaptureVideo(
    initial_world_map=root.world_map.load_or(cg.ARKitCreateScene()),
    label="Record the walkthrough",
)
session.clip &= capture.video
session.poses &= capture.poses
root.world_map &= capture.world_map
Source code in capturegraph-lib/capturegraph/procedures/nodes/collection/arkit/capture.py
@make_procedure
class ARKitCaptureVideo(Procedure[PosedVideo]):
    """Record a world-anchored posed RGB-D video with ARKit world tracking.

    The dense, continuous counterpart to ``ARKitCaptureScene``. Where a scene
    scan keeps a sparse, sharpness-gated *set of posed keyframes*, this records
    one continuous clip and bakes the 6-DoF camera pose into **every** frame,
    plus a best-effort metric depth map on the frames a depth sensor reaches
    (LiDAR/depth-camera devices). Depth is sparser than the RGB stream, so it is
    kept keyed by the exact frame it was measured at — the result is honest about
    which frames carry fresh depth rather than holding a stale map forward.
    Finishing archives the session's ``ARWorldMap`` the same way
    ``ARKitCaptureScene`` does, so the video shares a scene's coordinate frame
    and later sessions relocalize into it and extend the capture.

    The result is a ``PosedVideo``: the clip, its per-frame ``poses`` and
    ``depth`` (both keyed by frame instant so they line up), and the refreshed
    ``world_map``. Reach into the result's fields to store or forward them —
    ``capture.video``, ``capture.poses``, ``capture.depth``, ``capture.world_map``.

    Attributes:
        initial_world_map: The scene's world map to relocalize into. Wire it as
            ``root.world_map.load_or(cg.ARKitCreateScene())`` — the stored map when
            the scene exists, or a brand-new scene iff nothing is stored yet.
            (Named for what it is at session start, so ``capture.world_map`` stays
            free to navigate the result's refreshed map.)
        dynamic_range: Whether the session engages ARKit's video HDR —
            exposure fusion within the unchanged 8-bit BT.709 feed, better
            highlight retention — when the AR video format offers it
            (default: ``DynamicRange.SDR``); best-effort, see
            ``DynamicRange``.
        format: The frame-rate floor / resolution priority (default:
            ``PosedVideoFormat.RESOLUTION``) — see ``PosedVideoFormat``
            for the exact semantics of each case (no slow-motion case: ARKit
            tracking tops out at 60 Hz).

    Example:
        ```python
        capture = cg.ARKitCaptureVideo(
            initial_world_map=root.world_map.load_or(cg.ARKitCreateScene()),
            label="Record the walkthrough",
        )
        session.clip &= capture.video
        session.poses &= capture.poses
        root.world_map &= capture.world_map
        ```
    """

    initial_world_map: Input[ARWorldMap]
    dynamic_range: Input[DynamicRange] = DynamicRange.SDR
    format: Input[PosedVideoFormat] = PosedVideoFormat.RESOLUTION

ARKitScanScene #

Bases: Procedure[ARWorldMap]

Scan a world-anchored scene into an ARWorldMap without capturing images.

The registration-only counterpart to ARKitCaptureScene: the user walks the scene while ARKit extends its world map, and finishing archives the map alone — no keyframes, no depth, no clips. A study seeds each scene with one scan so the map (a few tens of megabytes) reaches the server before any image-bearing capture exists; later sessions relocalize into it immediately instead of queueing behind an image-upload backlog. The capture UI reads ARKit's live mapping status as a coverage indication, encouraging the user to keep scanning until the map is fully established.

Attributes:

Name Type Description
initial_world_map Input[ARWorldMap]

The scene's world map to relocalize into and extend. Wire it as root.world_map.load_or(cg.ARKitCreateScene()) — the stored map when the scene exists, or a brand-new scene iff nothing is stored yet.

Example
root.world_map &= cg.ARKitScanScene(
    initial_world_map=root.world_map.load_or(cg.ARKitCreateScene()),
    label="Scan the scene",
)
Source code in capturegraph-lib/capturegraph/procedures/nodes/collection/arkit/capture.py
@make_procedure
class ARKitScanScene(Procedure[ARWorldMap]):
    """Scan a world-anchored scene into an ``ARWorldMap`` without capturing images.

    The registration-only counterpart to ``ARKitCaptureScene``: the user walks
    the scene while ARKit extends its world map, and finishing archives the map
    alone — no keyframes, no depth, no clips. A study seeds each scene with one
    scan so the map (a few tens of megabytes) reaches the server before any
    image-bearing capture exists; later sessions relocalize into it immediately
    instead of queueing behind an image-upload backlog. The capture UI reads
    ARKit's live mapping status as a coverage indication, encouraging the user
    to keep scanning until the map is fully established.

    Attributes:
        initial_world_map: The scene's world map to relocalize into and extend.
            Wire it as ``root.world_map.load_or(cg.ARKitCreateScene())`` — the
            stored map when the scene exists, or a brand-new scene iff nothing
            is stored yet.

    Example:
        ```python
        root.world_map &= cg.ARKitScanScene(
            initial_world_map=root.world_map.load_or(cg.ARKitCreateScene()),
            label="Scan the scene",
        )
        ```
    """

    initial_world_map: Input[ARWorldMap]