18.1 Labels Are the Only Truth You Own

Every model in this book consumes labels, and every claim ultimately rests on them. Chapter 8's detection AP numbers, chapter 9's HOTA scores, chapter 12's ball-tracking precision, chapter 16's event taxonomy — all of them are claims relative to labels. The uncomfortable fact: labels are made by humans (or agents) and humans disagree. The professional discipline is not "labels are ground truth" — it is measuring whether your labels agree, and publishing the number. This chapter is the workflow that produces defensible labels: what to label, how many, with what tool, and with what agreement standard. The kappa protocol is source-backed (Cohen 1960, Landis and Koch 1977); the agreement numbers are measured (our lab's dual-annotation run, experiments/c18-annotation).

The workflow has four stages, and the order is load-bearing. Stage one, raw footage: the clip enters with provenance — pb-003 arrives as 900 frames at 25 fps from a fixed elevated camera with a SHA-256 hash; nrl-001 arrives as broadcast PTZ footage with pan drift measured at 9.76 px/frame (E08). Stage two, frame-level labels: geometry first (court or field keypoints — cheap, and for a fixed camera it amortizes to a one-time cost), then player boxes and identities, then the expensive small target (the ball). Stage three, event labels: contact-anchored events that inherit everything below them — an event label that cannot name the ball position and actor it rests on is an opinion, not a label. Stage four, quality review: a dual-annotated panel, an agreement metric per label family, adjudication of disagreements, and publication of the number. Skipping stage four is how datasets become folklore.

The two sports stress the workflow differently. Pickleball's fixed camera makes geometry nearly free and concentrates cost in the 8–14 px motion-blurred ball. Rugby league's broadcast camera makes geometry a per-frame cost (keypoints cannot be labeled once — the camera moves) and concentrates cost in player identity: 13.2 persons per frame on average, 24 at peak, and E08 measured 295 spawned track IDs where the true number of players is 26. Same workflow, different expensive stage.

18.2 The Tool Landscape (2026)

The tool decision is a scale-and-sovereignty decision, not a feature checklist. Six options cover the realistic space:

Three details matter more than the table. First, CVAT's Track Mode is the single biggest labor saver in the open lane: you label keyframes and the tool linearly interpolates boxes between them (K toggles a keyframe, O marks an object outside or occluded, merge reconnects fragmented identities), which vendor and community guides credit with 5–10× less manual work than frame-by-frame boxing [verify in lab]. The catch on Apple Silicon: CVAT is a multi-service Docker stack (server, UI, Postgres, Redis, OPA, ClickHouse) and not all images are native arm64 — components run under Rosetta x86 emulation [verify current image matrix]. Second, Label Studio's ML backend is the agentic hook: any local model can pre-fill tasks over a plain HTTP API, which is exactly the agent-drafts-human-disposes workflow of section 18.8, and the whole thing is pip install label-studio — pure Python, native arm64, no Docker. Third, Roboflow's free tier makes public-tier datasets public and grants content rights — read those terms before uploading unblurred club footage; for the book's amateur-sport regime that is disqualifying (section 18.9).

The license story is clean in 2026: CVAT is MIT and Label Studio is Apache-2.0, so the open lane has no AGPL-shaped trap — contrast chapter 8's Ultralytics licensing problem. Annotation tools are a solved purchasing decision; annotation labor is not. Choose by task: boxes/keypoints with heavy interpolation → CVAT; schema-heavy + review flows + agent pre-fill → Label Studio; error-driven iteration → FiftyOne alongside either. For the pb-003/nrl-001 regime — one practitioner, two sports, a few thousand frames, data that must stay local — the book's default is Label Studio with a local ML backend, CVAT when track interpolation dominates.

18.3 What to Label: The Five Label Families

Before budgets and tools, the taxonomy. Every label in this book is one of five families, and each family has its own cost profile and (section 18.5) its own agreement metric:

• Boxes — axis-aligned rectangles around players, referees, paddles. The cheapest spatial label (two corners), the native input of every detector in chapter 8, and agreed by IoU. Weakness: a box does not localize a player on the court plane — the feet do that, which is why the pb003 schema adds a foot point to every box.
• Masks — per-pixel silhouettes. Ten to thirty times the labor of a box and unnecessary for anything in the pb-003/nrl-001 pipeline; they earn their cost only when occlusion boundaries matter (chapter 10's segmentation lane) or when training a SAM-class model. Skip until a downstream model consumes them.
• Keypoints — named points: 12 court corners/line-ends in pickleball, 10 m-line intersections in rugby, ball centers, foot points. Agreed by distance in pixels. The fixed-camera gift: pickleball court keypoints are labeled once and propagated; rugby field keypoints must be re-labeled per sampled frame because the broadcast camera pans.
• Tracks — boxes plus identity over time. This is the expensive family: identity is a judgment call under occlusion, and every identity error compounds downstream (chapter 9's HOTA counts them). The rugby MOT ground truth is the heaviest labeling task in the book.
• Events — time-anchored classifications (serve, dink, contact; tackle, play-the-ball, kick). Agreed by Cohen's kappa. Events inherit every label below them and add two failure modes of their own: boundary placement (which frame is the contact?) and class ambiguity (dink or drop?), which is why the codebook — not the annotator — is usually what needs fixing.

Label exactly what downstream models train on and metrics consume — nothing more. The per-sport targets:

18.3b Two Worked Annotation Tasks

Pickleball, pb-003: the three-class task. The clip is 900 frames at 25 fps from a fixed elevated camera over a 13.41 × 6.10 m doubles court. The labeling task is deliberately small: player (four boxes, identity slots A1/A2/B1/B2 — a slot, never a name — plus the foot point the homography consumes), ball (a center coordinate plus a five-state visibility enum: visible / partial / blurred / occluded / not_visible), and paddle (an optional pilot tier, the RacketVision six-point paddle pose, ~100 frames to size the real cost of equipment-level annotation before committing, arXiv:2511.17045 [verify]). Three classes, but the schema carries more than three fields: the visibility enum, the actor and confidence on every event, and the annotators[] agreement block. The smallness is the lesson — a well-scoped three-class task with a measured agreement number beats a twenty-class task with none.

Rugby league, nrl-001: the 28-class task. The broadcast clip covers a 100 × 68 m field from a panning PTZ camera. The schema needs 25 player identity slots (thirteen-a-side minus the never-full overlap the clip actually shows; E08 measured 13.2 persons per frame on average, 24 at peak), plus ball, referee, and goalposts — 28 classes in total. Every design parameter is harsher than pickleball: far-end players are tens of pixels tall (the SAHI tiling of chapter 8 applies at label time too, because you cannot label what you cannot see); field keypoints must be re-labeled per sampled frame against a measured 9.76 px/frame pan drift; and track identity is the dominant cost — E08's detector spawned 295 IDs over the clip where 26 humans exist, which is precisely the disease the labeled set exists to measure. The deliverable format is MOTChallenge CSV (frame, id, bb_left, bb_top, bb_w, bb_h, conf, 1-based frames), feeding HOTA evaluation directly. E14's own next step is on record: if HOTA is claimed, label 300 frames first.

18.4 The 500-Frame Math

500 frames is the recurring budget because it is the smallest sample where a pickleball ball detector goes from noise to useful: the E04 log says "label 500 frames, fine-tune TrackNetV4". The number rests on a throughput model, and the model's dominant variable is whether a model pre-fills the annotation:

The pb-003 ball budget, single annotator: 500 frames × 10 s in the verify lane ≈ 83 minutes; fully manual at 30 s/frame ≈ 4.2 hours. Add the 10% dual-annotated gold panel (+25 min), player boxes pre-filled by E09 tracks (500 × 6 s ≈ 50 min), and events across the clip's contacts (15–30 min), and the full pb003-v1 label set lands at roughly 3–5 person-hours with agent pre-fill, 8–12 without. A weekend, not a grant. The pre-fill multiplier (~3×) is itself a measurable result the lab publishes.

Precision is a separate question from count. Human placement jitter on an 8–14 px blurred ball is the floor — a trained annotator with a zoom UI places the center with σ ≈ 1.5–2.5 px [verify in lab — measure it in the dual panel, don't assume]. Consensus of k independent annotators shrinks the error by √k, so ±1 px effective ground truth needs k ≈ 4, or k = 2 plus a sub-pixel intensity-centroid snap. But the model rarely needs ±1 px: TrackNet-family training consumes Gaussian heatmaps several pixels wide, and E10 evaluates at a 50 px tolerance while measuring 1.42 px RMSE against human labels. Set the precision budget from the metric the label feeds — and buy precision with consensus depth on a small panel, not with more frames. k = 4 on 50 frames calibrates your ground truth's noise floor better than k = 1 on 500.

The limit nobody escapes: annotation is the real cost of this entire discipline, and full-density labeling does not scale. A 30-minute match at 25 fps is 45,000 frames; the event layer alone — tagged at 3–4× real time, dual-annotated, adjudicated — costs 8+ hours before a single box is drawn, and frame-level labeling of the same match is hundreds of hours. Nobody labels a season. The only honest paths are the ones this chapter prices: a golden fixture that anchors everything, a 500-frame proof-of-value budget, a 10% dual panel for the agreement number, and active learning to spend the rest where the model is worst.

18.5 Inter-Annotator Agreement: Kappa, IoU, Distance

A label set without a measured agreement number is an opinion. Three label families, three agreement metrics:

• Categorical labels (events, shot types, visibility states): Cohen's kappa for two raters — observed agreement corrected for chance, κ = (p_o − p_e) / (1 − p_e). Three or more raters: Fleiss' κ; ordinal labels or missing data: Krippendorff's α. Raw percent agreement lies — with binary visibility calls, a 90% raw agreement can hide κ ≈ 0.4.
• Boxes: mean pairwise IoU plus match-rate at IoU ≥ 0.5 (PASCAL convention) — target ≥ 0.5 per box, ≥ 0.75 typical. Caveat: IoU measures task easiness as much as annotator skill — big centered players agree by default, so report per-class and per-size, never one headline number.
• Keypoints: distance in pixels (mean / median / p95) — target ≤ 3 px for ball centers. Pixels are the unit the downstream metric consumes; keep the agreement statement in the unit of consequence.
• Tracks: fragmentation count and ID switches between annotators; HOTA only once both annotators produce full tracks — chapter 9's machinery defines the vocabulary.

Landis & Koch (1977, Biometrics 33(1):159–174) band the scale: <0 poor, 0–0.2 slight, 0.2–0.4 fair, 0.4–0.6 moderate, 0.6–0.8 substantial, 0.8–1.0 almost perfect. Teach the caveat in the same breath: those bands are the authors' stated opinion, not an empirical law, and they are routinely over-read. For this book: κ ≥ 0.61 ("substantial") is the gate for an event class entering the training taxonomy; κ < 0.41 means the class definition is ambiguous and goes back to the codebook — low kappa is a specification bug before it is an annotator bug.

Our lab ran the protocol for real (experiments/c18-annotation/outputs/metrics.json): annotator A was the manual contact review of the six pb-003 golden frames; annotator B was a second rater with 2 of 6 deliberate disagreements. The measured result:

The fragility lesson: at n=6, two disagreements move kappa a full band. This is why the honest protocol is dual-annotation on 10%+ of frames — 90 of 900 for pb-003 — blind, codebook in hand, with rally-clustered bootstrap confidence intervals (chapter 20), aimed at kappa ≥ 0.61 per label family. Disagreements resolve by adjudication, never averaging: averaging two ball positions manufactures a point often more wrong than either when the ball is blurred, while an adjudicated decision grows the codebook's borderline examples. And "the label is the ground truth" is never true before the agreement number is published — the pb003 schema enforces this by refusing validation for a release whose annotators[].agreement block is empty.

18.6 Active Learning: Make Every Label Count

Random frame sampling wastes the budget on frames the model already handles. Active learning (Settles, Active Learning Literature Survey, 2009) spends it where the model fails. The loop: label a small random seed (~100 frames) → train the current model → infer over the unlabeled pool and score each frame by uncertainty (least-confidence, margin, or entropy; for detection, low max-score or disagreement across augmentations) → label the top-k worst frames → retrain. The honest stopping rule is the plateau on the gold panel, not a round number.

Two book-specific samplers sharpen the loop. Model disagreement: frames where the per-frame detector (E04 lineage) and the temporal model (E10) disagree about ball presence are maximally informative — each model's floor is the other's ceiling. The mid-confidence band: pre-fill confidence near 1.0 goes to the cheap auto-verify lane; near 0 is usually an invisible ball (cheap enum set); the middle band is where human time goes. And because events are class-imbalanced by nature — dinks dominate kitchen rallies, lets are rare — the labeling queue oversamples rare classes or the fine-tune inherits the imbalance (chapter 8's class-balance discipline, applied at label time). The budget allocation the book names: ~80% of frame throughput goes through the verify lane, ~20% through the full label lane (the 10% dual panel plus active-learning hard cases). The gold panel is not overhead — it calibrates the verify lane by measuring how often the pre-fill is right. At 500 frames the uncertainty spread concentrates labels where the model struggles; the effective sample is 3–4× larger than random sampling. FiftyOne is the tool for the inspection half.

18.6b Ten Use Cases: The Annotation Playbook

Sections 18.1–18.6 teach the workflow concept by concept. This section applies it: ten concrete annotation use cases in three named categories — frame-level labeling, time and event labeling, and quality/economics — each with the practical problem, the mechanism and its math, a figure, the pickleball and rugby league application, and the payoff. Every number carries its evidence label: measured (E-series or C18-lab), source-backed (paper), or [verify].

Category A: Frame-Level Labeling (UC 01–03)

UC 01 — The Three-Class Bootstrap: Player, Ball, Paddle (pb-003)

A club coach wants serve-quality analytics from one fixed camera and one 900-frame clip. The entire model stack of chapters 8–12 is blocked on one artifact: a labeled training set. The mechanism is the smallest schema that feeds the pipeline — four player identity slots (A1/A2/B1/B2) with foot points, one ball center with a five-state visibility enum (visible / partial / blurred / occluded / not_visible), one optional paddle tier. The math is the throughput model of 18.4: 500 frames × 10 s in the verify lane ≈ 83 minutes against 4.2 hours fully manual, and placement precision on the 8–14 px blurred ball is bounded by human jitter σ ≈ 1.5–2.5 px [verify in lab]. The visibility enum is the load-bearing detail: E04 teaches the ball is invisible in a large share of frames, and a forced coordinate on an invisible ball manufactures false ground truth that the fine-tune then learns.

UC 02 — The 28-Class MOT Set: Identity Is the Expensive Class (nrl-001)

A pathway analyst wants tackle-count and ruck-speed metrics from broadcast rugby league footage. Chapter 9's tracker fragments — E08 measured 295 spawned track IDs where the true player count is 26 — and the fix on record is E14's own next step: label 300 frames of MOT ground truth first. The mechanism is a 28-class schema: 25 player identity slots plus ball, referee, and goalposts, exported in MOTChallenge CSV (1-based frames) so HOTA consumes it directly. The math is brutal arithmetic: 300 frames × 13.2 persons per frame (E08 mean, 24 at peak) ≈ 3,960 box-plus-identity decisions, and every identity judgment under occlusion is a judgment call, not a measurement. Field keypoints must be re-labeled per sampled frame against the measured 9.76 px/frame pan drift (E08) — geometry never amortizes on a moving camera.

UC 03 — Mask Annotation for a SAM Fine-Tune

Chapter 10's segmentation lane needs masks only when occlusion boundaries matter — a tackle pile, a paddle edge against the net cord. Masks cost 10–30× the labor of a box, so the mechanism is a pilot, not a campaign: label 50–200 polygon masks [verify — practitioner pilot scale, no published benchmark], fine-tune a SAM-class model (chapter 10), then let the model propose masks and the human corrects boundaries. The math is the fine-tune cost curve: annotation minutes = N × (30–90 s per polygon) against the box rate of 3–8 s — the 100-mask pilot is 1–2.5 hours, and it answers one question: does the mask lane improve the downstream metric enough to justify 10–30× per frame?

Category B: Time & Event Labeling (UC 04–05)

UC 04 — Event-Level Annotation: Contact → Rally → Set

Frame labels answer "where is the ball"; coaches ask "who won the rally and why". Event labels are a hierarchy, and the mechanism is contact-anchoring: every event row names the contact instant, the actor slot, and the ball evidence it rests on — an event that cannot name its anchoring labels is an opinion. The math is inheritance plus two failure modes: events inherit every label below them, and add boundary error (which frame is the contact — ±1 frame at 25 fps is ±40 ms) and class ambiguity (dink or drop?). The taxonomy is the pb003 12-event enum (serve / return / third_shot_drop / third_shot_drive / dink / speedup / lob / volley / fault / let / kitchen_foot_fault / contact), and the conservative-closure rule applies: unknown unless contact and ball evidence support the call.

UC 05 — Video Timeline Annotation: Label the Span, Not the Frame

Tagging a 30-minute match frame-by-frame is hundreds of hours; tagging it on a timeline is the difference between possible and abandoned. The mechanism is video timeline segmentation (Label Studio's template is the open-lane default): the annotator marks spans — rally, tackle, play-the-ball — against a filmstrip and audio waveform, and the audio onset detector (chapter 15) drafts contact candidates with a ±frame window. The math is the real-time multiplier: span tagging runs at 3–4× real time dual-annotated and adjudicated, so the event layer of a 30-minute match costs 8+ hours before a single box is drawn — the number that kills "label the whole season" plans in 18.4. E06 measured the reviewable spans at 8.33 s and 12.83 s on pb-003, which sizes the timeline granularity.

Category C: Quality, Agents & Economics (UC 06–10)

UC 06 — Quality Review: The Kappa Panel

A label set without a measured agreement number is an opinion, and the practical problem is that nobody wants to pay for the second annotator. The mechanism is the 10% dual panel: two annotators independently label the same random 90 of 900 pb-003 frames, blind, codebook in hand, and the trio is computed per label family — Cohen's κ = (p_o − p_e) / (1 − p_e) on categorical fields, mean IoU + match-rate at ≥ 0.5 on boxes, distance-px on points. The C18-lab ran the protocol for real on the six golden frames: observed 0.667, expected 0.444, κ = 0.400 — Landis-Koch "fair" (measured, experiments/c18-annotation/outputs/metrics.json). The fragility math: at n=6, two disagreements move kappa a full band, which is why the gate is κ ≥ 0.61 on a 90-frame panel with rally-clustered bootstrap CIs (chapter 20), never a headline number on six frames.

UC 07 — Semi-Supervised Annotation: Model-Assisted Prefill

The single largest speedup available to a solo practitioner is not a better tool — it is never creating a label from scratch. The mechanism is schema-constrained prefill: every pipeline stage this book has built drafts its own fields (E10 temporal-diff drafts ball centers and pre-sets low-confidence frames to blurred/not_visible — measured 0.814 precision @50 px, 87.2% recall; E11 calibration drafts the 12 court keypoints at 3.82–4.65 cm RMSE; E09 BoT-SORT drafts player boxes and slots at HOTA 0.814), and the human disposes each field with exactly three actions: accept, correct, unknown. The math is the disposition distribution itself: accept-rate × 2 s + correct-rate × 10 s prices the verify lane, and the C18 throughput model puts the speedup at ~3× [verify in lab — the measured distribution is lab 1's deliverable]. E07 is the hard boundary: generative models draft prose, never labels.

UC 08 — The Active Learning Loop

Random frame sampling wastes the budget labeling frames the model already handles. The mechanism is the Settles (2009) loop: label a ~100-frame random seed, train, infer over the unlabeled pool, score each frame by uncertainty (least-confidence, margin, or entropy; for detection, low max-score or cross-augmentation disagreement), label the top-k worst, retrain — and stop at the plateau on the gold panel, not at a round number. The math is information per annotator-second: at 500 frames the uncertainty spread concentrates labels where the model struggles, making the effective sample 3–4× larger than random sampling [verify — lab 4 measures the AP gain against the gold panel]. Two book-specific samplers sharpen the queue: frames where the per-frame detector (E04) and the temporal model (E10) disagree about ball presence, and rarity weighting so rare classes (lets, kitchen foot faults) are not drowned by dinks.

UC 09 — Annotation Budget Planning: Cost per Frame

Every annotation plan dies in a spreadsheet before it dies in a tool, so the use case is the budget model itself. The mechanism: cost = N × t per lane, with t set by whether a model prefills — ball center + visibility is 20–40 s/frame manual against 5–15 s/frame verified, player box + foot is 15–30 s against 3–8 s, events are 30–90 s against 10–30 s [verify in lab — lab 3 replaces these practitioner norms with measured min/frame]. The math for pb003-v1: 500 × 10 s ≈ 83 min ball verify + 50 min pre-filled player boxes + 15–30 min events + 25 min dual panel ≈ 3–5 person-hours with agent prefill, 8–12 without. The limit nobody escapes: a 30-minute match is 45,000 frames at 25 fps, and full-density labeling does not scale — the honest budget is golden fixture + 500-frame proof-of-value + 10% dual panel + active learning on the rest.

UC 10 — The Review Gate: How Many Samples to Trust

The last use case is the one that decides whether the other nine shipped anything trustworthy: how large must the review sample be before the label set can be believed? The mechanism is the review gate — a random 10% panel, dual-annotated, with every field's agreement computed and the release refused when the schema's annotators[].agreement block is missing. The math is sampling error, not vibes. Consensus of k independent annotators shrinks placement error by √k (σ ≈ 1.5–2.5 px [verify], so ±1 px ground truth needs k ≈ 4, or k = 2 plus sub-pixel centroid snap), and the prefill acceptance rate is a binomial proportion: 90 panel frames give a Wilson 95% interval of roughly ±10 points around the observed accept rate — tight enough to gate the verify lane, too tight to skip. The rule the book names: precision is bought with consensus depth on a small panel (k = 4 on 50 frames calibrates the noise floor), and coverage is bought with the active-learning queue — never the reverse.

Runnable skeleton. The gate arithmetic is thirty lines of Python — compute κ, price the budget, size the panel:

def cohen_kappa(a, b):
    """a, b: parallel lists of categorical labels from two raters."""
    labels = sorted(set(a) | set(b))
    n = len(a)
    p_o = sum(x == y for x, y in zip(a, b)) / n
    p_e = sum((a.count(l) / n) * (b.count(l) / n) for l in labels)
    return (p_o - p_e) / (1 - p_e)

def budget_hours(n_frames, sec_verify=10, sec_manual=30):
    """The 18.4 throughput model: verify lane vs manual lane."""
    return n_frames * sec_verify / 3600, n_frames * sec_manual / 3600

# pb-003: 500 ball frames -> (1.4h verify, 4.2h manual); gate at kappa >= 0.61
# panel size: 10% of 900 = 90 frames; k=4 annotators shrinks jitter by sqrt(4)=2x

What this adds to the pipeline. UC 01 produces the ball training set chapter 12's fine-tune consumes; UC 02 produces the MOT ground truth chapter 9's HOTA claims stand on; UC 03 seeds chapter 10's mask lane; UC 04–05 produce the event layer that chapters 16, 17, and 23 classify and tactically mine; UC 06 and UC 10 publish the agreement numbers that bound what every AP, HOTA, and RMSE figure in chapters 8–12 can mean; UC 07–09 are the economics that make any of it affordable. The use cases are the chapter's argument made executable: ground truth is a workflow, and the workflow has a price list.

18.7 Formats & Versioning

Formats are interchange; the schema is the contract; hashes are the provenance. COCO JSON (flexible; pycocotools is the eval standard from chapter 8), YOLO txt (fast, lossy — no per-instance attributes), CVAT XML (tool-native; preserves track identity across keyframes), MOTChallenge CSV (the rugby MOT ground-truth format, 1-based frames, feeding HOTA directly) — plus the book's dataset/pb003 JSON schema (v1: 12 keypoints, 4 player slots, ball with visibility enum, 12 events, the annotators[] agreement block). The pb003 JSON is the only store humans or agents write to; every other format is a generated, one-way derived view. Never hand-edit a derived export.

The versioning discipline has four rules. Every labeled frame carries its source clip's SHA-256 and its own frame hash — a label without a hash of the bytes it describes cannot be audited. The dataset carries a version field (pb003-v1, -v2, …) schema-validated on every commit. Labels live in git (the whole 900-frame set is megabytes of JSON); media stays out (the pb-003 frames are not redistributable at all — section 18.9). And every label change triggers the golden-fixture regeneration of section 18.10, so a label commit and a metric move are always traceable to each other.

18.8 Annotation for Agents

Agents change annotation in one direction: auto-label + human verify. The agent's job is to pre-fill the schema so the human's job shrinks from create to dispose. Each pipeline stage this book has already built drafts its own fields: E10's temporal-diff energy filter drafts ball centers and pre-sets low-confidence frames to blurred/not_visible (measured: 0.814 precision @50 px, 87.2% recall — right often enough to make verify ~3× faster than manual); E11's calibration drafts the 12 court keypoints once per camera setup (3.82–4.65 cm RMSE); E09's BoT-SORT tracks draft player boxes, foot points, and slot proposals (HOTA 0.814, 2 switches on pb-003 — on nrl-001 the drafts degrade with the 295-ID fragmentation, and the human's correction load is the measured difference between the two sports); chapter 15's audio onset detector drafts contact candidates inside E06's review windows (8.33 s / 12.83 s per span).

The disposition protocol is deliberately tiny. Every pre-filled field arrives with confidence and review_required: yes, and the human has exactly three actions per field: accept (draft stands), correct (new value, flagged human_correction), unknown (the conservative-closure escape hatch). The harness records which happened per field, per frame — that distribution is the pre-fill quality metric, and it feeds the next active-learning round. One hard boundary from E07: generative models draft prose, never labels. The local VLM hallucinated serves and lobs and ignored the requested structure, so identity names, coaching notes, and anything E07-shaped stay human-only. For an agent to draft into the label store it needs four things the pb003 schema already carries: JSON Schema validation on write, enum tables (no free text), evidence-refs citing the run that produced each draft, and the review state machine. That is the minimum contract any agent-facing label store must expose.

18.9 Ethics

Labels of people are personal data. Faces in amateur footage: blur or skip — never label a face without consent, and a bounding box around an unconsenting bystander is still processing their image, so the label store records the skip, not the person. Athlete consent is purpose-bound: footage captured for coaching review does not automatically license a published dataset or model training — consent language must name the label set and the model, not just the camera. The label dataset itself may be personal data even when pixels never leave the lab: identity-stable tracks and event histories linked to an identifiable athlete qualify under GDPR/APP-style regimes. Hence the schema-level discipline — slots, not names (A1/A2/B1/B2), and any slot-to-name mapping table stored separately with its own access controls. The dataset/pb003 decision — labels CC-BY-4.0, frames not redistributable (PPA broadcast rights) — is the model: publish metadata and frame hashes so others can verify, not identity-bearing media. Rugby pathway footage raises the bar further: minors require guardian-level consent (machinery in chapter 31).

18.10 The Label-Verify Pipeline

The discipline that keeps labels honest over time: labels are inputs, fixtures are outputs. The golden fixture is regenerated from the label store on every accepted label change — never hand-patched — and the regression harness then re-runs the pipeline stages against it: detector outputs vs labeled boxes (IoU gate), temporal-diff ball centers vs labeled centers (distance gate; E10's 1.42 px RMSE is the baseline to hold), event proposals vs labeled events (precision gate; E10's 0.814). A label change that moves a metric is a deliberate, recorded event; a code change that moves a metric without a label change is a regression. This prevents the silent failure: an annotator fixes 20 mislabeled ball centers, the fine-tune "improves" by 4 AP, and the prose cites a model improvement that is actually a ground-truth correction. With fixture regeneration, the improvement is attributed to the label commit, where it belongs. The claims register consumes the result: a metric row that cannot name dataset version + fixture hash is unpublished.

1. Label → validate (schema) → assign the golden fixture.
2. Dual-annotate 10% → compute kappa/IoU/distance → publish.
3. Below substantial: adjudicate disagreements, re-label, re-measure.
4. Ship the dataset with the agreement number in the README.
5. The fixture regression harness asserts detector outputs after EVERY label change.

18.11 The Labeling Recipe (Copy This)

1. Pick the tool by task: boxes/keypoints with interpolation = CVAT; schema-heavy + agent pre-fill = Label Studio; error-driven curation = FiftyOne alongside.
2. Label the golden fixture first: the 6 frames cover serve-to-termination; they anchor everything else.
3. Geometry before players, players before ball, everything before events: the court keypoints amortize; the events inherit.
4. Budget by proof-of-value: 500 frames for the ball (the E04 log), one-time for court keypoints, 300 for players; scale only where value is confirmed.
5. Never label a coordinate you cannot also label a visibility state for: the invisible ball is data, not a labeling failure.
6. Dual-annotate 10%: kappa at least 0.61 (substantial) per label family; IoU at least 0.75 for boxes; at most 3px for keypoints.
7. Below threshold, adjudicate: re-label, re-measure. The number is the gate: κ≥0.61 admits a label to the taxonomy; κ<0.41 is a codebook bug, not a rater problem — fix the definitions first.
8. Active learning on the rest: label the model's worst quartile; the effective sample triples. Stop at the gold-panel plateau, not at a round number.
9. Publish the agreement number with every dataset release (the pb003 README rule).
10. Agent-annotated labels: auto-label, human-verify 10%, report the agent-as-rater kappa. Generative models draft prose, never labels.

18.12 What I Would Measure Next

• Run Label Studio with a local ML backend (E10 ball proposals + E09 tracks) on pb-003 — replace the pre-fill speedup estimates with a measured accept/correct/unknown distribution.
• Two annotators on the 90-frame gold panel (10% of 900) — get a real kappa with bootstrap CIs, not the n=6 demonstration.
• Timestamped annotation sessions — replace every [verify] in the 18.4 throughput table with measured min/frame per lane, and price the 500-frame set honestly.
• Active learning loop: 100 labels via uncertainty vs random — measure the AP gain against the gold panel.

18.13 Sources

• CVAT (MIT) — github.com/cvat-ai/cvat; Track Mode docs — docs.cvat.ai. arm64 Docker status [verify current image matrix].
• Label Studio (Apache-2.0) — github.com/HumanSignal/label-studio; video timeline + ML-backend pre-labeling templates — labelstud.io.
• FiftyOne (Apache-2.0) — github.com/voxel51/fiftyone; Roboflow Annotate (SaaS, public-tier content terms) — roboflow.com/annotate.
• Cohen (1960), kappa [verify primary citation]; Landis & Koch (1977), Biometrics 33(1):159–174 — pubmed.ncbi.nlm.nih.gov/843571; Fleiss (1971) multi-rater; Krippendorff's α.
• Settles (2009), Active Learning Literature Survey — minds.wisconsin.edu/handle/1793/60660; uncertainty sampling lineage: Lewis & Gale (1994).
• RacketVision paddle-pose grammar — arXiv:2511.17045 [verify license].
• Lab: lab/w4_lab_annotation.py → experiments/c18-annotation/outputs/metrics.json (measured: kappa 0.400, observed 0.667, expected 0.444, n=6, Landis-Koch "fair"). Anchors: E04 (ball wall, 1,082 detections / 900 frames), E06 (review windows 8.33 s / 12.83 s), E08 (rugby 13.2 persons/frame, 24 max, 295 spawned IDs, 9.76 px/frame drift), E09 (BoT-SORT HOTA 0.814, 2 switches), E10 (temporal-diff 0.814 precision @50 px, 87.2% recall, 1.42 px RMSE), E11 (calibration 3.82–4.65 cm RMSE).