Skip to content

Containers#

A loaded target is made of two broadcasting containers — cg.Array and cg.Map — that project attribute access across every element. This is what turns analysis into vectorized attribute access.

cg.Array#

cg.Array is a homogeneous sequence whose attribute access broadcasts to all elements, NumPy-style (it wraps its items rather than subclassing list):

import capturegraph as cg

survey = cg.load("/path/to/MySurvey", Survey)
sessions = survey.sessions.values()      # a Map's values broadcast too

# Instead of [s.location.latitude for s in sessions]:
latitudes = sessions.location.latitude    # chains through nested objects

Indexing returns an element, a slice returns an Array:

sessions[0]        # one element
sessions[1:3]      # an Array

Transforms#

.map(fn) applies a function element-wise (a Map keeps its keys); .pmap(fn) does the same across the process's one thread pool, sized to the CPU count (with cg.parallelism(n): caps how many elements are in flight at once), which pays off when fn releases the GIL (decoding an image, NumPy, a subprocess). A pmap inside a worker runs sequentially, the outer pool being the parallelism. .map_leaves(fn) descends into nested arrays and transforms only the leaves.

labels = sessions.rating.value.map(lambda r: f"{r}/5")
tooltips = sessions.photo.pmap(lambda p: p.tooltip())

Element types#

A container knows its element type where one exists: Array[Image] indexes and iterates as Image, a Map's values() is an Array of its values, and map/pmap carry the function's return type, so a mapped array reaches a typed parameter unaided.

frames = capture.frames                 # Array[SceneFrame]
thumbnails = frames.image.map(cgui.thumbnail)  # Array[Thumbnail]

A broadcast (frames.image) or a projection (frames["image"]) names a field whose type no checker can compute, so its elements are Any: the call that follows types them, as map(cgui.thumbnail) does above.

NumPy#

.to_numpy() converts to an array (Missing/None become nan):

import numpy as np

ratings = sessions.rating.value.to_numpy()
print(np.nanmean(ratings))

cg.Map#

cg.Map keeps its keys and broadcasts attribute access over its values, so you can index by key or project a field:

sessions = survey.sessions

sessions.keys()                  # every key (e.g. capture times, as cg.Time)
sessions.values()                # the entries (an Array, broadcasts)
sessions.items()                 # (key, value) pairs

sessions[some_time]              # one entry by key, a datetime or a cg.Time
sessions.rating                  # broadcast a field over all entries

A typed cg.Map[K, V] holds its keys as values of the key scalar K: a cg.Map[cg.Time, V] key is a cg.Time, whose .value is the datetime. A raw key given to the map (a lookup, in, an assignment, the constructor) is converted to that scalar first, so sessions[noon] and sessions[cg.Time(noon)] are the same entry. A bare cg.Map({...}) built by hand has no key type and keeps the keys it is given.

Order#

A Map's and an Array's entries always enumerate in ascending lexicographic order of their on-disk name — the same rule on every platform, so keys(), values(), and PathValues/PathKeys agree across the Python reader, the iOS client, and the sync tool. Because a Map[Time]/Map[Date] key is stored as a fixed-width hexadecimal-microsecond folder and an Array index as a zero-padded number ([cg.Array.index_name(index)][capturegraph.Array.index_name]), lexicographic order is chronological / numeric order — so the last entry is the most recent session. Keys compare by their folder name too, so the largest key is the latest capture time:

latest = survey.sessions.values()[-1]                # the most recent session
newest = max(survey.sessions.keys(), default=None)   # its key, a cg.Time (None if empty)

A Map[String] key sorts by the key text itself.

Missing data#

A per-element lookup that fails yields cg.Missing for that element instead of crashing the whole projection:

moods = sessions.mood            # sparse field: some entries Missing

for mood in moods:
    if not cg.is_missing(mood):
        print(mood.value)

Missing absorbs further attribute access, so deep chains stay safe; .to_numpy() turns it into nan. To fill the gaps instead, | substitutes a default for every Missing/None leaf (recursing into nested Arrays):

ratings = sessions.rating.value | 0       # Missing entries become 0

A Missing element can be stored. An array element, a map value or a struct field that is Missing is written as the marker file <entry>.cg-missing holding its reason (the exception class name, then its message), beside its siblings, so an array keeps its numbering and a map its keys; it loads back as Missing carrying that reason. The marker is an analysis-side convention: a capture platform never writes one, and a leaf it did not capture is simply absent.

Failing loudly vs. absent data#

Every broadcast surface — attribute access, string/tuple projection (arr["x"], arr["a", "b"]), .map/.pmap/.map_leaves, and calling the elements (arr()) — distinguishes a programmer error from legitimately absent data. A surface raises when every element fails for a non-data reason (a typo, a wrong key, a broken .map lambda — an AttributeError, KeyError, ZeroDivisionError, TypeError, …), so a mistake in an analysis script fails loudly instead of yielding a silent all-Missing result. The raised exception chains from the first element's failure.

The one carve-out is absence: a field the schema declares but a given entry never captured loads as Missing carrying a FileNotFoundError. When that is the reason on every entry, the projection stays all-Missing — so photos.exif() over a column of never-captured images is Missing, not an error. Any present element (a partial success) likewise never raises; only sparse data keeps chaining.

Complete Example#

import capturegraph as cg
import numpy as np
import pandas as pd

survey = cg.load("/path/to/MySurvey", Survey)
sessions = survey.sessions

df = pd.DataFrame({
    "date": [instant.value for instant in sessions.keys()],
    "rating": sessions.rating.value.to_numpy(),
    "notes": list(sessions.notes.value),
})

print(f"Sessions: {len(sessions)}")
print(f"Mean rating: {np.nanmean(df['rating']):.1f}")

See Also#