Sync Usage#

Installation#

CaptureGraph is not published on PyPI — the sync tool installs from a clone of the repository as part of the uv workspace (see Installation):

git clone https://github.com/CaptureGraph/CaptureGraph.git
cd CaptureGraph
uv sync

This provides the capturegraph-sync command (run it via uv run capturegraph-sync, or activate .venv first).

Basic Usage#

The server URL is a positional argument — there is no configuration file. To see what a server offers:

capturegraph-sync https://capture.example.org --list

Then sync a target:

capturegraph-sync https://capture.example.org --target MyStudy --output ./data

This:

1. Connects to the CaptureGraph server
2. Fetches the target's procedure definition and walks its schema
3. Downloads new and changed files to ./data/MyStudy/, in parallel, behind a live display — an overall meter (files, bytes, transfer speed, ETA) over one progress bar per in-flight file
4. Skips unchanged files and already-synced sessions

Interrupting a sync (Ctrl-C) is safe: everything already downloaded is recorded, and rerunning the same command resumes where it left off.

The server URL is forgiving: if you omit the scheme, capturegraph-sync adds https:// for a remote host and http:// for localhost. When it finishes, it prints a two-line summary:

✓ MyStudy synced in 3.4s
  12 downloaded (48.2 MB)  · 118 unchanged · 2 missing · 36 entries skipped

Command-Line Options#

capturegraph-sync SERVER_URL [OPTIONS]

The three --dng-to-* flags are mutually exclusive — passing more than one exits with an error. Running with neither --target nor --all-targets prints guidance, lists the available targets, and exits non-zero.

How a Sync Works#

Each target sync starts from the target's procedure definition, then runs in two phases:

Fetch the definition. The server returns the target's procedure definition {schema, root, nodes} (GET /api/v1/targets/{target}/definition). Its embedded schema is the on-disk layout, so once files land in their positions the synced folder loads with the data framework immediately — no separate index file is written.

1. Scan. The schema is walked recursively to build the full list of positions to check. Dynamic Map and Array directories — sessions and file sequences — are enumerated through the server's directory-listing endpoint (GET /api/v1/targets/{target}/directories, .../directories/{path}); all folders at one depth are listed concurrently, so a study with hundreds of sessions costs a few rounds instead of one request per folder. Any entry folder that already exists locally is skipped wholesale (unless --refresh-entries is set) — its files are never even resolved.
2. Resolve and download concurrently. The job list is sent to the server's leaf-resolve endpoint in chunks (the batch cap bounds each), every entry carrying the locally stored content version (if any). The server answers per leaf — unchanged, present, or missing. Only present leaves are downloaded, and they download in parallel across a thread pool (--jobs, default 16), pipelined behind the resolve so bytes start flowing while later batches are still being resolved. Each file streams in chunks — its bar moves while it transfers — and transient failures (connection drops, gateway errors) are retried with backoff. A small leaf arrives inline with the verdict, avoiding a separate round trip. Each finished download is applied on the orchestrating thread — the version stored, the progress advanced — keeping the version store single-threaded.

The resulting layout for a target whose schema has a sessions map:

./data/MyStudy/
└── sessions/                       ← one folder per Map/Array field
    ├── 00063B40E29D696A/           ← one folder per session (16-hex-digit ID)
    │   ├── photo.dng
    │   └── notes.json
    └── 00063B41A0F2C18D/
        └── ...

A local version store lives inside the output directory alongside this tree.

Version Store#

Sync maintains a local store inside each target's output directory, with one row per file, written in autocommit fashion — an interrupted sync never loses the versions of files already on disk.

• Versions are opaque, computed server-side. The client sends the versions it knows and stores what the server reports back; it never computes one itself (see content version).
• Keys are stored without file extensions, so a .dng that was later converted to .jpg still matches and is not re-downloaded.
• Deleting the store is safe but not free: on the next run the server reports every file without a known version as present, so files outside already-synced entry folders are re-downloaded once while the store repopulates. Existing entry folders are still skipped by folder existence.

Entries and --refresh-entries#

Map and Array entry folders (a session is the common case) are skipped by folder existence, not by content version: once an entry's folder exists locally, nothing inside it is checked on subsequent runs. This keeps repeat syncs of large deployments fast, but it means files that change inside an already-synced entry are invisible to a normal sync.

capturegraph-sync https://capture.example.org --target MyStudy --refresh-entries

Version Control#

Every synced target folder is written with its own .gitignore ignoring everything except itself and the generated schema.py loader — captured data doesn't belong in version control, and this keeps it out without modifying the enclosing repository.

Interactive Offers#

One situation prompts a question instead of acting silently. It is skipped (auto-declined) under --quiet or when stdin is not interactive.

• Leftover files (default: No). After syncing, files under the target folder that its current schema doesn't account for — leaves from an older schema, sessions deleted on the server — are listed with an offer to delete them. Sync never deletes anything on its own, and a target that synced with errors is left untouched.

Interceptor Files#

Some schema positions are filled by server interceptors (declared with @cgserver.intercept in the target — see Server Targets), which run per-user. capturegraph-sync sends no user identity, so the server reports those positions as missing: they show up in the summary's "Files missing on server" count instead of downloading. Per-user clients such as the iOS app send a request profile (their user id, and optionally their location) and do receive interceptor-generated files (delivered as dynamic leaves).

DNG Conversion#

Capturing RAW photos produces large .dng files. The --dng-to-jpeg, --dng-to-png, and --dng-to-heif flags convert them after all targets finish syncing, writing .jpg, .png, or .heic next to the original and deleting the .dng on success:

capturegraph-sync https://capture.example.org --all-targets --dng-to-jpeg

Conversion uses Apple Core Image on macOS and a rawpy/OpenCV pipeline elsewhere. It runs in multiple worker processes and makes several passes, skipping any output that already exists and is newer than its source — so a partial run resumes cleanly.

Working with Synced Data#

A synced folder is a regular capture target whose layout is its schema. Load it with cg.load, reusing the schema class the procedure was authored with:

import capturegraph as cg


class Session(cg.Struct):
    notes: cg.String
    rating: cg.Number


class Survey(cg.Struct):
    sessions: cg.Map[cg.Date, Session]


survey = cg.load("./data/MyStudy", Survey)

# Iterate sessions by date
for date, session in survey.sessions.items():
    print(date, session.notes.value, session.rating.value)

# Or broadcast a field across every session for vectorized analysis
all_notes = survey.sessions.notes.value

See Loading Captures for the full data framework.

Troubleshooting#

The CLI is built to catch the common mistakes before they turn into a stack trace — most of the cases below print guidance and the list of available targets, then exit non-zero.

No target chosen#

Sync needs either --target NAME or --all-targets. When neither is given, it prints guidance followed by the list of available targets — pick one and rerun.

Unknown target#

If --target names a target the server doesn't have, the CLI reports it, suggests the closest available name (Did you mean …?), and lists everything on offer. A --list run shows the exact names, including nested ones like Plants/Sunflower.

Connection errors#

There is no configuration file; the server URL is the positional CLI argument, so check it directly. On a failure the CLI prints an actionable hint (for example, a TLS error against an https:// URL suggests trying http:// for a local server):

• The scheme is optional — https:// is assumed for a remote host, http:// for localhost. Include the port if the server is not behind a reverse proxy (the server default is 4433)
• Run with just the URL and --list to confirm the server is reachable and see its targets

"Batch resolve failed"#

A failed batch request skips that entire chunk for the current run and is recorded as a single error in the summary. Nothing is corrupted — rerun the sync once the server is reachable again.

A file re-downloads on every run#

Check that the local version store still exists in the target's output directory — if it was deleted (or the output path changed), versions are unknown and non-session files are fetched again once while the store rebuilds. If the store is intact, the file's content genuinely changes on the server between runs, so each sync sees a new version.

New captures don't appear in existing sessions#

Plain syncs skip any entry whose folder already exists. Run with --refresh-entries to resolve files inside existing entry folders.

Exit codes#

capturegraph-sync exits with code 1 if the --dng-to-* flags conflict, the server is unreachable, a named target does not exist, no target is selected, or any target fails to sync; 130 when interrupted (progress is kept — rerun to resume); otherwise it exits 0.

See Also#

• CaptureGraph Sync Overview — How sync works
• Data Framework — Loading and analyzing synced data
• Loading Captures — Using cg.load
• Running a Server — The server that sync talks to