Skip to content

gpu

gpu #

gpu_run — a driver script run in the GPU interpreter, never in the library's own.

The interpreter is the one CG_GPU_PYTHON names, else the one ns-train on the path belongs to: torch and every model package live there. A driver is a capturegraph-free script run by path with its arguments, under whichever lock the caller holds; gpu_lock is the one every GPU-bound recipe takes, one holder per device, so jobs on different GPUs of one machine run at once.

GPU_PYTHON_ENV = 'CG_GPU_PYTHON' module-attribute #

Names the interpreter holding torch and the GPU model packages.

gpu_lock() #

Hold the GPU this process's drivers run on, one holder per device.

The device is the first of CUDA_VISIBLE_DEVICES (the first GPU when it is unset), counted in PCI-bus order as nvidia-smi and every driver count it, and the lock is named after its UUID, so holders of two GPUs of one machine never wait on each other. A machine without nvidia-smi has the one lock "gpu".

Raises:

Type Description
CalledProcessError

If nvidia-smi knows no such device.

Source code in capturegraph-recipes/capturegraph_recipes/gpu.py
def gpu_lock() -> AbstractContextManager[None]:
    """Hold the GPU this process's drivers run on, one holder per device.

    The device is the first of ``CUDA_VISIBLE_DEVICES`` (the first GPU when it
    is unset), counted in PCI-bus order as ``nvidia-smi`` and every driver
    count it, and the lock is named after its UUID, so holders of two GPUs of
    one machine never wait on each other. A machine without ``nvidia-smi``
    has the one lock ``"gpu"``.

    Raises:
        subprocess.CalledProcessError: If ``nvidia-smi`` knows no such device.
    """
    return cg.lock(_device_lock_name())

gpu_python() #

The interpreter drivers run in.

Raises:

Type Description
ScratchError

If CG_GPU_PYTHON is unset and ns-train is not on the path.

Source code in capturegraph-recipes/capturegraph_recipes/gpu.py
def gpu_python() -> Path:
    """The interpreter drivers run in.

    Raises:
        cg.scratch.ScratchError: If ``CG_GPU_PYTHON`` is unset and ``ns-train`` is not on the path.
    """
    named = os.environ.get(GPU_PYTHON_ENV)
    if named:
        return Path(named)
    ns_train = shutil.which("ns-train")
    if ns_train is not None:
        return Path(ns_train).with_name("python")
    raise cg.scratch.ScratchError(
        f"No GPU interpreter: set {GPU_PYTHON_ENV} to a python with torch and the "
        f"model packages installed, or put its ns-train on the PATH."
    )

gpu_run(work, driver, argv, *, env=None) #

Run driver with argv in the GPU interpreter, work as its working directory.

The child inherits the process environment with torch's settings and env merged over it; its output streams into work's exec.log. The interpreter runs with -P, keeping the driver's own directory off sys.path, so a driver may share its name with the package it imports.

Raises:

Type Description
ScratchError

If no GPU interpreter can be found.

CalledProcessError

If the driver fails; the exception carries the tail of its log.

Source code in capturegraph-recipes/capturegraph_recipes/gpu.py
def gpu_run(
    work: cg.ManagedDir,
    driver: Path,
    argv: Sequence[str | os.PathLike[str]],
    *,
    env: Mapping[str, str] | None = None,
) -> None:
    """Run ``driver`` with ``argv`` in the GPU interpreter, ``work`` as its working directory.

    The child inherits the process environment with torch's settings and
    ``env`` merged over it; its output streams into ``work``'s ``exec.log``.
    The interpreter runs with ``-P``, keeping the driver's own directory off
    ``sys.path``, so a driver may share its name with the package it imports.

    Raises:
        cg.scratch.ScratchError: If no GPU interpreter can be found.
        subprocess.CalledProcessError: If the driver fails; the exception
            carries the tail of its log.
    """
    work.exec(
        [gpu_python(), "-P", driver, *argv],
        env={**os.environ, **_TORCH_ENV, **(env or {})},
    )