Skip to content

The Procedure JSON Format#

A serialized procedure is the interoperability standard of CaptureGraph — not the Python classes, and not any one client. The Python recorder is one way to author this JSON; the iOS app is one way to execute it. Any device or frontend that speaks the format can execute procedures to the best of its abilities and contribute captures to the same dataset: a client only implements the node kinds and types its procedures actually use, so a new platform (an Android app, a kiosk, another authoring DSL) can participate without knowing the full vocabulary.

This page specifies the format so new implementations can be built directly against it. Its sibling, The Captured-Data Format, specifies the other half of the standard: the wire contract a client uses to sync the files a procedure produces.

Top-level format#

A serialized procedure has three keys:

{
  // the target's data layout — the recursive type wire form
  "schema": { /* type object */ },

  // id of the node where execution begins
  "root": "B",

  // every node in the graph, keyed by id
  "nodes": {
    "0": { /* node object */ }
  }
}

schema is fixed a priori: it is the shape of the dataset the procedure fills, so analysis can be written before the first capture and a server knows every file that can exist. There is no separate runtime manifest — the schema travels with the procedure.

Node ids are uppercase hexadecimal strings of uniform width, assigned children-first, so the root has the highest id. Ids are local to one document and carry no meaning beyond linking nodes.

The schema#

The schema is the type wire form: each type is an object with a "type" tag. Every scalar is reverse-DNS namespaced under edu.cornell.* and self-describing — it carries its kind ("json" or "binary") and extensions so a reader without the lib can store or skip its values ({"type": "edu.cornell.image", "kind": "binary", "extensions": [...]}). The structural constructors stay bare: struct carries fields, map carries key and element, array carries element. A field whose name starts with _ (such as _metadata) is hidden and lands under a leading . on disk (.metadata).

An enum scalar (e.g. edu.cornell.video_format) is an ordinary scalar on the wire; the closed case set it admits is part of each platform's type implementation, cross-checked by the golden wire vectors like the rest of the vocabulary.

Naming#

Every identifier that crosses the wire follows one rule-set, so a reader can predict a name without consulting an implementation:

  1. Lowercase snake_case everywhere serialized — JSON keys, enum case strings, wire-type-name segments, file stems, storage columns. Only external standards keep their own casing (EXIF tag names, OpenEXR channels, PLY properties, HTTP headers such as ETag).
  2. Reverse-DNS names the byte-format owner. Formats CaptureGraph coins live under edu.cornell.* (embedded-video metadata keys under edu.cornell.video.*); vendor-opaque blobs live under the vendor (com.apple.arkit.world_map). Multi-word segments use _. One identifier never names two byte layouts.
  3. A node kind is its class name on every platform: CamelCase with a category prefix, verb-first for collection nodes (CaptureImage, ARKitCreateScene); operators are singular Type + Operation, carrying the result's unit when dimensioned (LocationDistanceMeters, TimeDifferenceSeconds).
  4. Inputs and settings use a fixed role vocabulary: source is read or navigated from; destination is the path written to; value is the stored/literal payload (and a unary operator's operand); default is a fallback value; procedure is a wrapped child and body a loop's; left/right are symmetric operands, primary/fallback an asymmetric pair; condition gates. A node's single subject is named by its type (image), role-qualified only beside a second of the same type (reference_image). User-shown strings are text/*_text; UI choice lists are options, data-keying label lists are labels.
  5. Dimensioned fields carry a unit suffix (_meters, _seconds, _degrees, _mps, _dps, _ev, _kelvin, _lumens, _celsius, _hpa, _mmph; dimensionless fractions are _ratio). Exempt, documented once: WGS84 latitude/longitude (inherently degrees); the shared rigid-pose / pinhole-camera block (quaternion_*, position_*, focal_*, center_*, reference_*, *_field_of_view — metres, pixels, degrees), identical across Angle, Pose, and the embedded video intrinsics; and epoch instants, always named time and carried as epoch seconds.
  6. Closed case sets are enum scalars with lowercase snake_case case strings. A vendor passthrough (Weather.condition, verbatim from WeatherKit) is documented as free-form and exempt.
  7. No local exception: platform code that persists anything — caches, queues, settings — derives the same snake_case keys (the Swift @SnakeCaseCoding/@CGJSONScalar macros generate them from property names), so a copy-pasted struct cannot drift.

Node object format#

{
  // class name of the node; the executing client resolves behavior from it
  "kind": "CaptureImage",

  // wire type of the value this node produces ({"type":"void"} = side effect).
  // A scalar self-describes: its edu.cornell.* name carries kind + extensions.
  "return_type": { "type": "edu.cornell.image", "kind": "binary", "extensions": ["heic", "dng", "jpeg", "jpg"] },

  // optional human-readable label shown to the user
  "label": "Session photo",

  // optional plain-JSON configuration
  "settings": { "field": "primary" },

  // optional named data-flow edges: input name -> producing node id
  "inputs": { "source": "7" },

  // optional ordered control-flow children, each returning {"type":"void"}
  "substeps": ["4", "A"]
}

Only kind and return_type are always present; a null label and empty settings/inputs/substeps are omitted rather than written empty. inputs are data flow (the node consumes the referenced nodes' values), substeps are control flow (run in order for their side effects), and settings are inert configuration.

return_type is a compact type object. The path family returns {"type": "path"} — a non-storable cursor into the schema; storage nodes return the scalar type they write.

Example#

This procedure:

import capturegraph as cg


class Session(cg.Struct):
    primary: cg.Image


class Survey(cg.Struct):
    reference: cg.Image
    sessions: cg.Map[cg.Date, Session]
    _metadata: cg.Metadata


@cg.procedure(Survey)
def survey(root):
    root.reference |= cg.CaptureImage(label="Reference photo")
    session = root.sessions[cg.CaptureTime()]
    session.primary &= cg.CaptureImage(label="Session photo")


json_str = survey.to_json()

serializes to:

{
    "schema": {
        "type": "struct",
        "fields": {
            "reference": {
                "type": "edu.cornell.image",
                "kind": "binary",
                "extensions": [
                    "heic",
                    "dng",
                    "jpeg",
                    "jpg"
                ]
            },
            "sessions": {
                "type": "map",
                "key": {
                    "type": "edu.cornell.time",
                    "kind": "json",
                    "extensions": [
                        "json"
                    ]
                },
                "element": {
                    "type": "struct",
                    "fields": {
                        "primary": {
                            "type": "edu.cornell.image",
                            "kind": "binary",
                            "extensions": [
                                "heic",
                                "dng",
                                "jpeg",
                                "jpg"
                            ]
                        }
                    }
                }
            },
            "_metadata": {
                "type": "struct",
                "fields": {
                    "_thumbnail": {
                        "type": "edu.cornell.thumbnail",
                        "kind": "binary",
                        "extensions": [
                            "jpeg"
                        ]
                    },
                    "_location": {
                        "type": "edu.cornell.location",
                        "kind": "json",
                        "extensions": [
                            "json"
                        ]
                    },
                    "_next_notification": {
                        "type": "edu.cornell.time",
                        "kind": "json",
                        "extensions": [
                            "json"
                        ]
                    }
                }
            }
        }
    },
    "root": "E",
    "nodes": {
        "0": {
            "kind": "GetRoot",
            "return_type": {
                "type": "path"
            }
        },
        "1": {
            "kind": "PathField",
            "return_type": {
                "type": "path"
            },
            "settings": {
                "field": "reference"
            },
            "inputs": {
                "source": "0"
            }
        },
        "2": {
            "kind": "Constant",
            "return_type": {
                "type": "edu.cornell.image_encoding",
                "kind": "json",
                "extensions": [
                    "json"
                ]
            },
            "settings": {
                "value": "heic"
            }
        },
        "3": {
            "kind": "CaptureImage",
            "return_type": {
                "type": "edu.cornell.image",
                "kind": "binary",
                "extensions": [
                    "heic",
                    "dng",
                    "jpeg",
                    "jpg"
                ]
            },
            "label": "Reference photo",
            "inputs": {
                "encoding": "2"
            }
        },
        "4": {
            "kind": "CacheProcedure",
            "return_type": {
                "type": "edu.cornell.image",
                "kind": "binary",
                "extensions": [
                    "heic",
                    "dng",
                    "jpeg",
                    "jpg"
                ]
            },
            "settings": {
                "skip_if_exists": true
            },
            "inputs": {
                "destination": "1",
                "value": "3"
            }
        },
        "5": {
            "kind": "RequiredProcedure",
            "return_type": {
                "type": "void"
            },
            "inputs": {
                "procedure": "4"
            }
        },
        "6": {
            "kind": "PathField",
            "return_type": {
                "type": "path"
            },
            "settings": {
                "field": "sessions"
            },
            "inputs": {
                "source": "0"
            }
        },
        "7": {
            "kind": "CaptureTime",
            "return_type": {
                "type": "edu.cornell.time",
                "kind": "json",
                "extensions": [
                    "json"
                ]
            }
        },
        "8": {
            "kind": "PathKey",
            "return_type": {
                "type": "path"
            },
            "inputs": {
                "source": "6",
                "key": "7"
            }
        },
        "9": {
            "kind": "PathField",
            "return_type": {
                "type": "path"
            },
            "settings": {
                "field": "primary"
            },
            "inputs": {
                "source": "8"
            }
        },
        "A": {
            "kind": "Constant",
            "return_type": {
                "type": "edu.cornell.image_encoding",
                "kind": "json",
                "extensions": [
                    "json"
                ]
            },
            "settings": {
                "value": "heic"
            }
        },
        "B": {
            "kind": "CaptureImage",
            "return_type": {
                "type": "edu.cornell.image",
                "kind": "binary",
                "extensions": [
                    "heic",
                    "dng",
                    "jpeg",
                    "jpg"
                ]
            },
            "label": "Session photo",
            "inputs": {
                "encoding": "A"
            }
        },
        "C": {
            "kind": "CacheProcedure",
            "return_type": {
                "type": "edu.cornell.image",
                "kind": "binary",
                "extensions": [
                    "heic",
                    "dng",
                    "jpeg",
                    "jpg"
                ]
            },
            "settings": {
                "skip_if_exists": false
            },
            "inputs": {
                "destination": "9",
                "value": "B"
            }
        },
        "D": {
            "kind": "RequiredProcedure",
            "return_type": {
                "type": "void"
            },
            "inputs": {
                "procedure": "C"
            }
        },
        "E": {
            "kind": "ProcedureSequence",
            "return_type": {
                "type": "void"
            },
            "label": "survey",
            "substeps": [
                "5",
                "D"
            ]
        }
    }
}

Read the graph as the recorder built it. GetRoot is the cursor at the schema root; PathField/PathKey walk it (reference, then sessions[<time>].primary). Each CaptureImage produces an image, and a CacheProcedure writes that value to the destination path — skip_if_exists: true for |=, false for &=. Each cache is wrapped in a RequiredProcedure, the Void step the ProcedureSequence actually runs (so a stored value is never left dangling); the cache returns its value type so it can also be read back. The JSON carries no trace of how it was authored — which is exactly what makes it the standard.

Loops#

A ProcedureWhile node carries an extra template_nodes setting: the ids of nodes the executing client clones afresh for each iteration. The serializer guarantees loop hygiene — no node outside the loop references a template node, and nested loops' templates stay contained in their enclosing loop's.

Executing a procedure#

A client processes the graph by:

  1. Looking up each node's behavior from its kind string
  2. Topologically sorting via inputs and substeps edges
  3. Executing from root, evaluating a node's inputs before the node and its substeps in list order
  4. Resolving path nodes against the on-disk layout the schema describes, and writing cached values there

The node vocabulary is open-ended. The authoritative list of kinds, with their input/setting signatures, is introspectable via build_node_schema — the same schema the management panel's node editor is built on (that editor is currently deferred). The path family, CacheProcedure, and the unified Constant are lowering internals and are flagged hidden from the editor palette, but they are ordinary nodes on the wire.