Skip to content

parallel

parallel #

The one thread pool every pmap runs on, and the parallel map rule.

  1. Pool. One process-wide pool, sized to the CPU count. A pmap made from inside a worker runs sequentially, the outer pool being the parallelism, so nested fan-out never oversubscribes and never waits on the pool it occupies.
  2. Cap. parallelism(workers) bounds how many elements each pmap in the block has in flight at once; the default is the pool's size.
  3. Map. pmapped is array.mapped across the pool: the same results and failure rule, a logged count of the elements as they finish with one worker entry per pool thread beneath it, each element one turn of its worker; every element runs in a copy of the caller's context so the active scratch, workspace and log scope carry over. The first element to raise propagates as soon as it finishes, and elements not yet started are cancelled.

POOL_SIZE = os.cpu_count() or 1 module-attribute #

How many threads the pool runs, and the default cap on elements in flight.

parallelism(workers) #

Cap how many elements each pmap in the block has in flight at once.

with cg.parallelism(4):
    thumbnails = sessions.photo.pmap(session_thumbnail)

Parameters:

Name Type Description Default
workers int

The cap; 1 runs every pmap in the block sequentially.

required
Source code in capturegraph-lib/capturegraph/types/containers/parallel.py
@contextmanager
def parallelism(workers: int) -> Iterator[None]:
    """Cap how many elements each ``pmap`` in the block has in flight at once.

    ```python
    with cg.parallelism(4):
        thumbnails = sessions.photo.pmap(session_thumbnail)
    ```

    Args:
        workers: The cap; ``1`` runs every ``pmap`` in the block sequentially.
    """
    token = _in_flight.set(max(workers, 1))
    try:
        yield
    finally:
        _in_flight.reset(token)

pmapped(items, function, label, on_error) #

mapped across the pool: the parallel rule Array and Map share.

Parameters:

Name Type Description Default
items Iterable[I]

The elements to apply function to.

required
function Callable[[I], R]

The per-element operation.

required
label str | None

What the count shows; function's name when None.

required
on_error OnError

What an element that raises does to the walk.

required

Returns:

Type Description
list[R]

What mapped returns, in order.

Source code in capturegraph-lib/capturegraph/types/containers/parallel.py
def pmapped[I, R](
    items: Iterable[I],
    function: Callable[[I], R],
    label: str | None,
    on_error: OnError,
) -> list[R]:
    """``mapped`` across the pool: the parallel rule ``Array`` and ``Map`` share.

    Args:
        items: The elements to apply ``function`` to.
        function: The per-element operation.
        label: What the count shows; ``function``'s name when ``None``.
        on_error: What an element that raises does to the walk.

    Returns:
        What ``mapped`` returns, in order.
    """
    from capturegraph.types.containers.array import mapped

    items = list(items)
    if _inside_pool.get() or len(items) < 2:
        return mapped(items, function, label, on_error)
    with progress(total=len(items), label=label or label_of(function)) as counted:
        workers = _Workers(on_error)
        pool = _shared_pool()
        cap = _in_flight.get()
        futures: list[Future[R]] = []
        running: set[Future[R]] = set()
        try:
            for item in items:
                while len(running) == cap:
                    running = _unsettled(running)
                context = copy_context()
                context.run(_inside_pool.set, True)
                future = pool.submit(context.run, workers.run, function, item)
                future.add_done_callback(lambda _: counted.advance())
                futures.append(future)
                running.add(future)
            while running:
                running = _unsettled(running)
        except BaseException:
            for future in futures:
                future.cancel()
            raise
        results = [future.result() for future in futures]
        return guard_broadcast(results, "pmap") if on_error == "missing" else results