The Captured-Data File Format#

A serialized procedure says what a target's data will look like — its schema is the data layout. This page specifies the other half of the interoperability standard: the wire contract by which a client moves captured files — the typed scalar leaves of that layout — to and from a server. Where the procedure JSON lets any frontend execute a procedure, this format lets any frontend sync its results: an Android client, a second command-line tool, or a browser can mirror a target's captures knowing only this contract.

The canonical models are the Pydantic classes in capturegraph.api_models — the batch request/response shapes, the caller-profile shape, and the X-CG-* header names and codecs — shared verbatim by the server (which produces them) and capturegraph-sync (which consumes them). This page is the language-neutral description of those shapes, plus the REST-shaped single-leaf routes that carry no body of their own.

Addressing a leaf#

Every captured value lives at a position in the target's schema, which maps to a relative path on disk. A file path is the chain of on-disk folder names from the target root to a leaf, with the leaf's own stem — its file name without extension — as the last element. There is no separate "name" field: the stem is the last path component.

["sessions", "00063B40E29D696A", "photo"]

Each component is one container step — a struct field, a map key, or an array index — carried as the folder name it resolves to. The schema-driven on-disk layout (a hidden _-field becomes a leading ., Map keys become key-encoded folders, Array indices become zero-padded folders) is specified once under Types and Nodes; a client producing a file path applies that layout, and the names travel here already encoded.

A file path travels in one of two ways, depending on the route:

• In the URL, for the single-leaf REST routes (streaming, saving, deleting, listing a directory): each component is percent-encoded and the components are joined with /, so the path above addresses .../files/sessions/00063B40E29D696A/photo.
• As a JSON array of strings, for the batch routes (resolving, saving many files at once): "path": [...] inside each entry.

The leaf's extension is never part of the path — a stem can itself contain dots (e.g. .metadata), so folding the extension into the address would be ambiguous. It travels separately: in the X-CG-File-Extension header on the raw-body routes, or as an extension field alongside path in a batch entry or leaf state. Reading a leaf never asks for a preferred extension either — the server serves whatever is actually on disk, since the schema fixes which scalar (and therefore which extensions are even valid) lives at that position.

Schema validation#

Every request path — reads and writes alike, single-leaf or batch — is resolved against the target's declared schema before it ever touches disk. Each component must be one of:

• a struct field, named by its on-disk form (a hidden _-prefixed field's on-disk name starts with a leading .);
• a Map key that re-encodes, through that key type's own codec, to exactly this folder name — a lenient decoder is not allowed to admit an alias folder name that shadows the canonical one;
• an Array index in its canonical zero-padded form.

A leaf path's final component must resolve to a scalar; stopping at a container (or asking for a leaf on a directory route) is invalid. On a write, two further checks apply: the claimed extension must be one the leaf's scalar declares, and the leaf must not be a path an interceptor owns.

Because this is an internet-exposed endpoint, confining every request to paths the schema actually declares means a file that happens to sit on disk but isn't part of the schema — a stray temp file, a private note dropped in by hand — is simply unreachable through the API: no request path resolves to it.

The caller profile#

Who (and optionally where) a request comes from travels as headers on every request — X-CG-User-Id and X-CG-Location — entirely outside file identity, so the same two headers apply uniformly whether the call is a single leaf or a batch covering thousands of them. The server reads them wherever a leaf could be interceptor-generated (streaming a leaf, resolving leaves) and otherwise ignores them.

• X-CG-User-Id — the requesting client's identity, so an interceptor can resolve per-user.
• X-CG-Location — the caller's opted-in position, four comma-separated decimals in a fixed order: latitude,longitude,altitude_meters,heading_degrees (the WGS84 fields of the edu.cornell.location value). A header that doesn't parse as exactly four decimals is a 400.

Both headers are optional and independent; an entirely anonymous request sends neither.

The content version#

A leaf carries one content version: an opaque token identifying its exact bytes. Two leaves with equal versions hold identical content; a leaf whose version differs from the client's cached one has changed. The server computes it as the base64-encoded blake2b digest of the bytes, but the client treats it as opaque — it only compares versions for equality and uses one as a cache key. A client never computes a version itself, so the digest algorithm can change without touching any frontend. A static file and an interceptor leaf both carry one.

The version rides differently depending on how the leaf travels: as the standard ETag header (quoted) on the raw-body streaming/saving routes, matched against a request's If-None-Match; as the version field on a LeafState everywhere else, including known_version in a resolve request.

Resolving leaves#

POST files:resolve is the incremental-sync workhorse: in one round trip a client asks, for a batch of leaves, whether each has changed since the version it holds. The request is a list of entries, each pairing a file path with the version the client currently has cached (null when it holds no copy):

{
  "entries": [
    { "path": ["sessions", "00063B40E29D696A", "photo"], "known_version": "x3Qf…" }
  ]
}

The response is a positional list — results[i] answers entries[i]:

{
  "results": [
    {
      "status": "present",
      "version": "y7Pa…",
      "extension": "heic",
      "dynamic": false,
      "inline_base64": null
    }
  ]
}

A single-leaf check is just a batch of one. The whole save/resolve API is built from one verdict object, the leaf state.

Batch size#

A resolve batch is capped at 4096 entries per request, the single source of truth for that number; a larger request is rejected with 422. The cap is generous — targets hold tens of thousands of tiny leaves and each entry is a few hundred bytes, so a full sweep completes in a handful of requests while a malformed client cannot post unbounded work.

The leaf state#

A LeafState is the server's verdict for one leaf, and the only payload of the resolve and save APIs. It has exactly one status:

The payload fields:

• version — the content version of the present bytes. Always set on present.
• extension — the leaf's actual on-disk extension (e.g. "heic"), one of the several its scalar may declare (an Image might be stored as .heic or .dng). Always set on present.
• dynamic — true for an interceptor leaf, whose value the server recomputes on every request. A client may cache one for offline use but must not enrol it in background revalidation, because its version is expected to drift. Defaults to false.
• inline_base64 — the leaf's bytes, base64-encoded, shipped inside the verdict to save a separate download. Present for small static files and always for dynamic leaves; absent otherwise (see inline bytes).

unchanged and missing carry no payload — none of version, extension, inline_base64, or dynamic appears. This is the single invariant a faithful implementation must hold on both sides: a present leaf names its content, and the other two are bare. The model enforces it, so a malformed verdict fails loud rather than silently mis-syncing.

A client resolving a batch reads each verdict as: unchanged → keep the cached file; missing → the leaf is absent server-side (delete a stale local copy, or count it as not-yet-captured); present → adopt version as the new cache key, then take the bytes from inline_base64 if set, otherwise fetch them from GET files/{path}.

Inline bytes#

To avoid a second round trip for tiny files, a present verdict may carry the bytes themselves in inline_base64. The rule, stated once here:

• A static file is inlined when its size is below the inline threshold of 1024 bytes (1 KiB) — the single source of truth for that number; larger files omit inline_base64 and are fetched from GET files/{path}.
• A dynamic (interceptor) leaf is always inlined — the server has just computed its bytes, so shipping them costs nothing extra and spares the client a follow-up request.

The threshold is deliberately small: a target can hold tens of thousands of tiny leaves, and inline bytes are dead weight whenever the client's copy turns out to be current (which unchanged already covers).

Where metadata travels#

One rule still governs the whole file API: when a message body carries raw file bytes, its accompanying metadata travels in HTTP headers; otherwise it travels in the JSON body.

• The two raw-body routes are single-leaf: streaming a leaf (GET files/{path}) returns the bytes as the response body, with extension and version in X-CG-File-Extension / ETag; saving a leaf (PUT files/{path}) sends the bytes as the request body, extension in X-CG-File-Extension.
• Every other call — resolving or saving a batch, listing or deleting a directory, deleting a single leaf — has a free body, so the path(s) and their metadata stay in readable JSON.

Orthogonally, the caller profile rides in headers on every request regardless of body shape — it is caller context, not message payload, so it doesn't follow the raw-bytes-vs-JSON split at all.

A leaf's address lives directly in the URL for every single-leaf route (REST-shaped), and its version rides the standard ETag/If-None-Match pair — there is no bespoke per-request header for either.

Streaming a leaf#

GET files/{path} streams a single leaf's raw bytes back (application/octet-stream). It is how a client fetches any leaf not delivered inline. Two response headers describe the bytes:

• ETag — the leaf's quoted content version.
• X-CG-File-Extension — its actual on-disk extension.

A request carrying If-None-Match: "<version>" gets a bodyless 304 Not Modified (with the same ETag) when that version is still current, instead of resending the bytes. A leaf that neither an interceptor claims nor exists on disk is a 404.

An interceptor leaf is served here too: the server computes it on the fly and streams the result with the same two headers, extension always "json".

Saving a leaf#

PUT files/{path} uploads one captured file as the raw request body — the bytes are streamed straight to disk, so the client can move the captured file into its upload queue with no copy. The extension rides in the required X-CG-File-Extension header. The response is a present leaf state for the stored leaf (small files carry their bytes inline too, the same rule as any present verdict), so an immediately following resolve of the same leaf reports unchanged with no extra request.

The path and extension are both checked against the schema before anything is written: an extension the leaf's scalar doesn't declare is a 422, and a save to a position an interceptor owns is refused with 409 — the server, not the client, is the source of truth for those leaves.

Saving many files at once#

POST files:save uploads many small files inline in one round trip — the batch counterpart to a single PUT, for a target's usual mix of thousands of tiny leaves rather than one large capture:

{
  "entries": [
    { "path": ["sessions", "00063B40E29D696A", "notes"], "extension": "json", "data_base64": "eyJ2IjogMjF9" }
  ]
}

Every entry is validated against the schema and decoded before anything is written — a single bad entry rejects the whole batch, so a client never sees a partial apply. Valid entries are then written, each file individually atomic, and answered with a present leaf state per entry, positionally matching the request.

The batch is capped, the single source of truth for these numbers: each entry is ≤ 64 KiB (the small-file boundary), the whole batch is ≤ 512 KiB of decoded bytes, and — as a flood backstop — ≤ 65536 entries. Because a client packs small files by total size, a batch of 10–20-byte files runs to thousands of entries, so the byte cap, not the count, is the real limit. A request over any cap is rejected with 422 before anything is written, and a file at or above the per-entry limit always goes through the single streaming PUT files/{path} instead.

Deleting#

• DELETE files/{path} deletes the leaf's on-disk file (stem.*) — {"deleted": bool} reports whether anything existed to remove.
• DELETE directories/{path} recursively deletes a schema directory and everything inside it — the same {"deleted": bool} response.

Both are refused with 403 while the server runs with client deletes disabled (a per-deployment setting; the management panel can always delete).

Listing directories#

GET directories (the target root) and GET directories/{path} list a schema directory's distinct child entry stems — subdirectory names plus file stems (extension stripped), sorted. The response is an EntriesResponse:

{ "exists": true, "entries": ["00063B40E29D696A", "00063B41A2C8F1B0"] }

exists is the directory's own presence on disk, told apart from an empty listing so a loaded Array/Map can tell a present-but-empty collection from an absent one. This generalizes Map (key-encoded folder names) and Array (zero-padded indices) enumeration in the schema's on-disk layout.

Interceptor leaves#

A schema position can be filled by a server-side interceptor rather than by an uploaded file — this is how a target schedules the next capture or validates a location (see Targets and Interceptors). On the wire an interceptor leaf is just a leaf with dynamic set:

• In a resolve verdict, or from GET files/{path}, it is present with dynamic: true, its freshly computed version, an extension of "json" (interceptor values are JSON scalars such as cg.Time or cg.Location), and — in the resolve verdict — its bytes always in inline_base64.
• A write to an interceptor-owned path is refused with 409 (see Saving a leaf).

A client consuming one stores the bytes like any captured file but, because the value is recomputed per request, must not poll it for change in the background — it is a live answer, not a settled artifact. A client that sends no profile (an anonymous sync) sees interceptor positions resolve as missing, since the interceptor has no user to compute for.

Known tech debt: percent-encoded / in map keys#

The server recovers a request's file path by taking the URL's path segment and splitting it on / — but the ASGI layer decodes percent-escapes on the whole URL before that split ever runs, so a %2F a client sends to escape a literal / inside one component is indistinguishable, by the time the server sees it, from an actual path separator. A Map keyed by a scalar whose canonical encoding can contain / — a plain String key is the obvious foot-gun — can end up with entries that are on disk and schema-valid but unaddressable through the REST file/directory routes. Prefer a key type whose canonical encoding never contains / until this is fixed.

Implementing a sync#

A frontend mirroring a target combines this format with the procedure JSON:

1. Fetch the definition {schema, root, nodes} from GET definition and walk its embedded schema to enumerate every leaf position. Dynamic Map/ Array directories are listed via GET directories/GET directories/{path}, which reports whether a directory exists and its child entry names.
2. For each leaf, build a file path and a known_version from local cache.
3. Resolve leaves in batches of at most 4096. Act on each verdict: unchanged → skip; missing → absent; present → write the bytes (inline if carried, else from GET files/{path}) and record version.
4. Treat version strictly as an opaque cache key — never recompute it — and skip background revalidation of any dynamic leaf.

The endpoints, all under /api/v1/targets/{target}, are summarized in Running a Server.