Skip to content

multiview

multiview #

multiview — a multi-view model's cameras, points and depths for a set of images.

  1. Run. As predict: weights first, then the staged images through the model's driver under the GPU lock, with --multiview so the set is posed together in one forward pass.
  2. Cameras. Each camera the driver reports (OpenCV camera-to-world at the staged size) becomes a Pose in the convention every reconstruction shares — OpenGL, y up, looking down -z — with its intrinsics re-expressed at the stored image's size, keyed by the image's position.
  3. Model. The cameras, the driver's points and an empty set of observations make a scale-free Reconstruction; the per-image depths are RelativeDepth values in that same scale.

multiview(images, model, max_dimension=1600) #

The cameras, points and depths model predicts for images posed together.

The model is scale-free: cgr.sfm.aligned carries it onto prior positions, or cgr.sfm.bundle_adjust refines it once tracks exist. Its points are a seed as they are.

seen = cgr.depth.multiview(images, cgr.depth.PI3X)
model = cgr.sfm.aligned(seen.model, frames.pose)

Parameters:

Name Type Description Default
images Array[Image]

The photographs to pose together.

required
model Model

The model to run.

required
max_dimension int

The longest side, in pixels, the images are staged at, which is the size the depths come back at.

1600

Raises:

Type Description
ScratchError

If no GPU interpreter can be found.

CalledProcessError

If the driver fails; the exception carries the tail of its log.

Source code in capturegraph-recipes/capturegraph_recipes/depth/inference/multiview.py
@cg.pure_function("depth.multiview/1")
def multiview(
    images: cg.Array[cg.Image],
    model: Model,
    max_dimension: int = 1600,
) -> Multiview:
    """The cameras, points and depths ``model`` predicts for ``images`` posed together.

    The model is scale-free: ``cgr.sfm.aligned`` carries it onto prior
    positions, or ``cgr.sfm.bundle_adjust`` refines it once tracks exist. Its
    points are a seed as they are.

    ```python
    seen = cgr.depth.multiview(images, cgr.depth.PI3X)
    model = cgr.sfm.aligned(seen.model, frames.pose)
    ```

    Args:
        images: The photographs to pose together.
        model: The model to run.
        max_dimension: The longest side, in pixels, the images are staged at,
            which is the size the depths come back at.

    Raises:
        ScratchError: If no GPU interpreter can be found.
        subprocess.CalledProcessError: If the driver fails; the exception
            carries the tail of its log.
    """
    weights = snapshot(model.weights)
    work = cg.ManagedDir.new()
    sizes = stage_images(images, work, max_dimension)
    started = time.monotonic()
    with gpu_lock():
        output = run_driver(work, model.driver, weights, multiview=True)
    cameras = {
        camera_key(index): _pose(camera, width, height)
        for index, (camera, (width, height)) in enumerate(
            zip(read_cameras(output), sizes, strict=True)
        )
    }
    positions, colors = read_points(output)
    cg.logging.info(
        "Predicted multi-view model",
        model=model.weights.repo_id,
        revision=model.weights.revision,
        images=len(images),
        points=len(positions),
        seconds=time.monotonic() - started,
    )
    return Multiview(
        model=Reconstruction(
            cameras=cg.Map[cg.String, cg.Pose](cameras),
            points=cg.PointCloud.of_points(positions, colors),
            observations=Observations.empty(),
        ),
        depths=read_depths(output, len(images)),
    )