The Movie Metadata Format (.mov embedded lens metadata)#

The byte-layout spec for the camera metadata CaptureGraph bakes into a recorded .mov, so any reader — a Python analysis pass, a third-party player, another platform's recorder — recovers it without reverse-engineering the writer. It completes the interoperability standard for video captures: the captured-data format moves the file, and this page says what is inside it beyond the pixels.

Container#

QuickTime .mov, written by an AVAssetWriter. Camera metadata rides in QuickTime metadata items (AVMetadataItem) in the mdta key space, in two scopes:

1. Asset-level snapshot — one item on the movie's metadata, a whole-clip average.
2. Per-frame timed metadata track — a .metadata track carrying one item per video frame at that frame's presentation time.

Both scopes use the same identifier and the same value encoding, so a reader decodes intrinsics identically wherever they appear.

Camera intrinsics#

CaptureGraph coins this key, so it lives under edu.cornell.* — the owner-of-bytes namespace rule the scalar vocabulary also follows. The video. segment scopes it to embedded-video metadata keys, kept apart from the scalar type-name space: edu.cornell.pose already names the Pose scalar, whose byte layout differs from this key's JSON payload.

The value is the JSON encoding of CameraIntrinsics — byte-identical to the JSON the still pipeline stashes in the EXIF user comment, so intrinsics are one format across a capture's stills and its movies. Its fields (sensor-native pinhole, in pixels, referenced to the buffer's landscape pixel grid):

{
  "focal_x": 3312.5,
  "focal_y": 3313.1,
  "center_x": 2015.7,
  "center_y": 1511.2,
  "reference_width": 4032,
  "reference_height": 3024,
  "distortion": null
}

focal_x/focal_y are the focal lengths (matrix diagonal); center_x/center_y the principal point (matrix third column); reference_width/reference_height the pixel dimensions of the frame the matrix was measured against. distortion is null for the per-frame track and the live snapshot (the live video stream carries no distortion model); it is populated only where a still resolves one. The matrix is not rotated into display orientation — it stays aligned to the sensor-native landscape buffer, so it composes with the video track's orientation transform, never double-applied.

Per-frame track timing#

Each frame's item is one AVTimedMetadataGroup over [pts, pts + duration), where pts is the video frame's presentation time in the movie timeline (the writer's session starts at the first frame's PTS) and duration is the frame's sample duration — so an entry's start equals the matching video frame's PTS. That shared PTS is the frame↔intrinsics correspondence. A frame the sensor delivered without an intrinsic matrix leaves a gap (never fabricated data); the track is present only when intrinsic-matrix delivery is active, so it is never empty. The .metadata track's format description declares the single identifier above with the UTF-8 data type.

Source#

The matrix comes from the kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix attachment AVFoundation delivers on each CMSampleBuffer when isCameraIntrinsicMatrixDeliveryEnabled — a matrix_float3x3, column-major: column 0 = (focal_x, 0, 0), column 1 = (0, focal_y, 0), column 2 = (center_x, center_y, 1). The writer reads it from the exact buffers it encodes (CameraIntrinsics(sampleBuffer:)), so the reading and the frame share a presentation time with no second read.

Camera pose (ARKit posed video)#

The ARKit posed-video recorder adds a second identifier to this same per-frame track for the 6-DoF world transform, keeping the intrinsics identifier and encoding above unchanged so both writers share one format description. Each frame's AVTimedMetadataGroup then carries two items — the intrinsics item above and the pose item below — over the same [pts, pts + duration) range, so a reader recovers intrinsics and pose for that frame at one presentation time.

world_transform is the camera's 4×4 world transform (ARKit world space, gravity-aligned) as sixteen doubles in column-major order — the layout simd_float4x4.columns.0…3 lay out, so the first four entries are column 0, and entries 12–15 are the fourth column (the translation (x, y, z, 1)). It is the same pose ARKit reports for the frame, so it is aligned to the sensor-native landscape buffer, never rotated into display orientation.

This baked-in pose is a self-contained bonus — a player recovers per-frame pose without the CaptureGraph sidecars. The schema-advertised truth stays the PosedVideo.poses map (Map[Time, Pose]), which additionally carries the decomposed position/quaternion, intrinsics, tracking state, and capture time per frame. Like the intrinsics track, a pose item carries no extendedLanguageTag: the metadata adaptor matches an item to a format-description entry by the full (identifier, dataType, extendedLanguageTag) tuple and raises an uncatchable NSException on a miss.

Sidecar alignment (the frame ↔ pose ↔ depth contract)#

The recorder guarantees, and readers may rely on:

1. One pose entry per video frame, atomically. A frame's video sample, its metadata items, its pose entry, and its depth entry (when the sensor measured one) are appended together or dropped together — a frame the writer skips (input not ready, failed append) leaves no artifact anywhere. So the i-th pose in key order is the i-th video frame in presentation order, for every i; readers decode a frame by counting, never by elapsed-time × average fps (the clip is variable-rate — ARKit drops frames under load).
2. Sidecar keys are affine in PTS. The session-clock → Unix-epoch offset is latched once per recording, so every sidecar key equals offset + PTS exactly: keys are strictly monotone in frame order and their deltas equal the movie-timeline deltas (immune to wall-clock steps mid-recording).
3. Depth keys are a subset of pose keys. A depth entry reuses the same key value as its frame's pose entry — frame↔depth correspondence is exact key equality, never a nearest-timestamp match.