10.1 Boxes Say Where; Masks Say What

A bounding box is a rectangle around something. A mask is the thing itself. For sports, the difference is decisive in three places: the paddle blade (you need the contour to compute the face angle), the ball (you need the silhouette to separate it from the court), and the player (you need the foreground to compute foot placement from the mask bottom). Segmentation is the tool, and 2024-2026 made it promptable — you ask the model for the object you mean, in language or a click.

A box commits two approximations a mask does not. First, it includes background and other objects inside the rectangle: at the pickleball non-volley zone, two players' boxes overlap heavily, and any crop taken from the box contains the opponent's legs and the net tape. In a rugby league tackle the problem compounds — the ball carrier's box contains the first defender's arms, the second defender's head, and a slice of the pitch, and chapter 8's detector measured 13.2 persons per frame on nrl-001 (E08), so nearly every box is contaminated by at least one other body. Second, the box's geometric summary is a fiction of the rectangle: the bottom-center point is the midpoint of the rectangle's lower edge, not a foot. On a wide ready stance at the kitchen line, or a defender's split-leg bracing posture in a tackle, the box bottom-center lands on air between the feet — and that is the exact point the chapter 6 homography (E05/E11: 3.82-4.65 cm RMSE) consumes.

A mask pays back exactly where those two approximations bite: foot-point extraction under wide stances, ReID crops without background contamination, and sub-box objects like the ball and paddle that a box localizes sloppily. Where neither approximation bites — a full-body player in open field feeding a tracker — the box is enough, and chapter 8's detector output is the cheaper signal. The rule of this chapter: masks are a precision instrument, deployed at named junctions, not a default upgrade.

Every fact in this chapter is measured (our lab, with experiment IDs), source-backed (Meta's releases, licenses read from the source), or [verify]-marked where a claim outruns the evidence.

10.2 The Segmentation Taxonomy

The field reuses four words loosely, and each implies different outputs, metrics, and pipeline roles. Pin them down before spending money:

The first three families are closed-vocabulary: the class list is baked into training, and a pickleball paddle or a rugby Steeden is simply not in it. The fourth is the 2023-2025 break: the prompt is the class definition, so an object the model never saw in training is segmentable on day one. Video adds three variants: VOS (video object segmentation — you supply the frame-0 mask, the model propagates it; semi-supervised), VIS (video instance segmentation — detect, mask, and track instances fully automatically), and SAM 3's PCS-video (detect, segment, and track every instance of a concept). SAM 2 is a VOS machine with a promptable front end; SAM 3 is a PCS machine. Neither is a VIS tracker in the chapter 9 sense — see 10.7 for why that distinction is load-bearing.

Metrics deserve the same precision. IoU (Jaccard) scores one mask: intersection over union against ground truth. mIoU averages per-class IoU over classes — the semantic-segmentation standard, with a sport trap: the dominant class (court, background) inflates the mean, so report per-class IoU for the rare class (ball, line pixels) or the number lies. Mask AP is the COCO instance-segmentation standard, averaged over IoU thresholds 0.50 to 0.95; its strictness is useful, because AP at high thresholds punishes exactly the sloppy boundaries that break foot-point extraction. Boundary IoU computes overlap only on a narrow band around the mask edge — it measures the contour where sport actually cares, since feet are a boundary phenomenon [verify: no published sport-segmentation benchmark reports Boundary IoU; candidate book contribution]. For video, J&F (DAVIS) pairs region IoU with boundary accuracy; SAM 2's published video numbers are J&F.

For the book's labs, mask quality on ball, paddle, and feet is measured with IoU plus Boundary IoU against roughly 50 hand-labeled frames; AP is reserved for comparisons against published instance baselines; mIoU appears only when a semantic court parse is the actual product — for example, deriving a pickleball court-surface mask as a calibration prior, or separating crowd from field in a broadcast rugby frame before line detection.

10.3 SAM 2: The Streaming Memory Architecture

Meta's SAM 2 (Ravi et al., 2024, arXiv 2408.00714, Apache-2.0) unified image and video promptable segmentation in one streaming model. It is this book's workhorse segmenter for a simple operational reason: it is the last SAM generation that runs natively on the M4 lab. Understanding its six components tells you exactly what it can and cannot promise.

1. Image encoder — Hiera, a hierarchical ViT. SAM 1's encoder was a plain ViT producing a single feature scale, with a heavyweight neck faking a feature pyramid on top. Hiera (Ryali et al., ICML 2023) is a pyramid: it is pretrained with masked autoencoding and progressively pools its tokens, so coarse late stages carry semantics while early stages retain spatial detail, and skip connections hand the mask decoder both. That single change is why SAM 2 segments images roughly 6x faster than SAM 1 [verify: Meta-reported figure] — the encoder stopped paying for an adapter on a single-resolution trunk. Four sizes ship (Hiera-T/S/B+/L, roughly 39M/46M/81M/224M parameters); SAM 2.1 (September 2024) refreshed all four checkpoints with retrained weights at identical sizes — use 2.1 everywhere. Lab default: Hiera-B+; Hiera-L when boundary quality on feet matters.

2. Prompt encoder — clicks, boxes, masks. Geometric prompts are embedded and injected into the decoder: a point is labeled foreground or background, a box delimits the object extent, and a prior mask refines an existing segmentation. One click on a pickleball paddle blade in frame 0 is a complete prompt; so is chapter 8's detector box around a rugby ball carrier, which makes the detector-to-SAM handoff a one-line glue layer.

3. Memory bank — two FIFO queues plus object pointers. This is the temporal heart. The bank holds (a) a FIFO of the N most recent frames' memories — short-term motion context, ordered by temporal position embeddings; (b) a FIFO of up to M prompted frames' memories — the model never forgets where you clicked, even after the recent-frames queue rolls past; and (c) object pointers: compact vectors distilled from the mask decoder's output tokens, carrying what the object is at a semantic level alongside the spatial maps. Critically, the bank stores mask-conditioned memories, not raw frames: after each prediction, a lightweight convolutional memory encoder fuses the predicted mask with the frame's unconditioned image features and pushes the result onto the bank. The past arrives pre-digested.

4. Memory attention — the frame queries its own past. Before decoding, the current frame's features cross-attend to the stored memories (and self-attend among themselves), conditioning the representation on where the object was and what it looked like. There is no recurrence, no optical flow, no state-space model — attention over a bounded memory is the entire temporal mechanism.

5. Mask decoder — streaming multi-mask output. The SAM-1-style two-way transformer decodes the conditioned features against the prompt tokens and emits several candidate masks plus a predicted IoU score for each. Under an ambiguous prompt — a click on a torso, which could mean player or jersey — the candidates are ranked by predicted IoU and the best wins. The same predicted-IoU score doubles as an automatic "I lost it" flag during video propagation: when it collapses, the mask has degraded, and your pipeline should distrust the geometry without any external supervision.

6. The streaming inference loop. Per frame, left to right: encode the frame once with Hiera; cross-attend to the memory bank; decode each prompted object's mask; fuse mask and features through the memory encoder; push onto the FIFO. History is never re-processed and memory stays bounded by the FIFO caps — the property that makes hour-long match footage tractable at all.

The memory-attention vs full-attention tradeoff is the design decision to understand. Full attention over all past frames costs O(T) memory and compute per new frame — hour-long video becomes quadratic in the total frame count and dies. SAM 2's bounded bank costs O(N+M) per frame: constant, cheap, streamable. What you surrender is perfect recall: an object that vanishes for longer than the FIFO window is re-anchored only by the prompted-frame memories and the object pointers — semantic similarity, not pixel continuity. In practice that trade is nearly all upside, because sport occlusions (a player crossing the net post, a ball carrier buried in a ruck for two seconds) are short. But it defines the failure mode: when the re-emergence region contains a similar object — the other pickleball player in identical kit crossing at the NVZ, or the second defender in the same jersey arriving at the tackle — memory attention can lock onto it with equal confidence. Nothing in the architecture knows these are two different people; similarity to stored memory is the only identity criterion. Section 10.7 builds the boundary rule on exactly this.

The occlusion-recovery story, stated as a mechanism: during occlusion, memory attention finds no matching evidence in recent frames and the mask degrades gracefully rather than hallucinating — the predicted IoU drops, which is your automatic loss signal. On re-emergence, the prompted-frame FIFO and object pointers re-anchor the object by matching against the original prompt memory, bridging appearance drift across the occlusion window semantically rather than by raw pixels. For a single salient object this is a genuinely stronger temporal prior than a Kalman filter.

Pickleball case. One click on the paddle blade in frame 0 of pb-003, and SAM 2 propagates the paddle mask through the entire rally — prompt-once, track-many. This is the structural cure for the E04 ball-detection wall (1,082 detections at median score 0.1125 — the frame-by-frame detector barely sees the ball): propagation matches on motion-conditioned appearance, and between consecutive frames the ball's mask overlaps itself even at 40 pixels per frame of flight, so smallness hurts detection but not propagation. The paddle — larger, slower, rigid — is the easier case and the lab's hands-on object.

Rugby league case. Prompt the ball carrier before a tackle on nrl-001 and SAM 2 tracks his mask into and through the ruck — 13-plus bodies, multi-second occlusion, camera pan. The memory bank's prompted-frame anchor is what recovers him on the far side. The failure to teach is the symmetric one: when two same-kit defenders converge on the carrier simultaneously, the mask can exit the tackle attached to the wrong body, and the predicted IoU does not always flag it, because the new region matches the stored memory almost as well. Region tracking survives the occlusion; identity does not.

The MPS story, honestly. SAM 2 is CUDA-first code. On Apple Silicon the official path is PyTorch MPS with PYTORCH_ENABLE_MPS_FALLBACK=1: unimplemented MPS ops fall back to CPU, and every fallback forces a GPU-to-CPU sync that can make MPS slower than pure CPU on some pipelines; Meta documents possible numerical drift versus CUDA [verify: community ledger, sam2 issue #462 — the lab's own SAM 2 wall-clock numbers are a 10.9 measurement, not yet run]. Practice: enable the fallback only if an op crashes, downsample input, keep tensors in memory, and treat SAM 2 video on M4 as offline batch, never real-time.

10.4 SAM 3: Concept Segmentation and the 2026 Reality

SAM 3 (Meta, November 19-20 2025, arXiv 2511.16719, ~848M parameters ≈ 1.7GB in bf16) changes the prompt from geometry to concept. Promptable Concept Segmentation: a text phrase — "player in white", "pickleball", "ruck" — or image exemplars produce detection, segmentation, and tracking of every matching instance through the video, with per-instance session IDs. SAM 2's question was "segment this thing I clicked"; SAM 3's is "segment everything matching this description." One prompt replaces per-object boxes — directly useful for rugby clusters where jersey color separates the teams, and for pickleball where "paddle" is a two-instance concept no detector class ever contained.

Three architectural facts matter. First, a shared Perception Encoder feeds both halves: a DETR-based detector conditioned on text, geometry, or exemplars finds and segments all instances per frame, and a memory-based video tracker inherited from SAM 2 propagates them — detector and tracker are decoupled, so tracking can be corrected interactively without re-running detection. Second, a learned presence token first answers "does this concept exist in the scene at all?" before localization, decoupling recognition from localization; this is the mechanism that separates near-identical prompts like "player in white" from "player in red" that a DETR head alone confuses — team-color and role phrases become first-class queries. Third, the benchmark it ships with, SA-Co, spans 270K unique concepts (over 50x prior open-vocabulary segmentation benchmarks); Meta reports SAM 3 at 75-80% of human performance on SA-Co, roughly doubling prior systems on image and video PCS [verify: Meta-reported; no sport-footage replication exists yet].

The caveats are the story. The official repo ships custom CUDA/Triton kernels (Triton NMS among them) with no official MPS path — production use means Colab GPU batch processing with artifacts synced back. Friction paths exist (the Ultralytics integration, API-driven but AGPL-3.0 — a copyleft landmine for shipped products; a Hugging Face transformers port), but neither is the supported lane. And the license is the custom SAM License — royalty-free with attribution required in publications, plus Trade Controls and no-reverse-engineering clauses — not Apache. Read it before publishing figures, not after.

SAM 3.1 (March 2026) is a drop-in efficiency update, not a new capability — same prompts, same backbone, no accuracy change. It fixes the exact problem rugby hits first: SAM 3 tracked each object independently, so 26 players plus ball plus referees meant roughly 28 sequential tracking passes per frame, cost scaling linearly with object count. Object Multiplexing groups tracked objects into buckets processed jointly through shared memory; Meta reports about 7x faster at large object counts (up to 128 objects on a single H100) [verify: vendor claim on vendor hardware; the book's only planned check is the Colab lab on nrl-001]. A 26-player rugby broadcast frame is precisely the regime where SAM 3 was impractical and SAM 3.1 is merely expensive. Pickleball — four players, ball, two paddles — never left the practical zone. The deeper lesson generalizes: per-object-loop architectures die on field sports; shared-computation architectures survive them.

10.5 DINOv3: The Foundation Behind the Meshes

DINOv3 (Meta, August 2025, arXiv 2508.10104) is a self-supervised vision backbone family — ViT-S/B/L/H+ up to a 7B flagship, trained on 1.7 billion unlabeled images with self-distillation, RoPE position embeddings, and SwiGLU activations, plus distilled ConvNeXt variants. No labels anywhere in training: the supervision signal is the images themselves. For sports this is the "frozen features + small head" recipe made concrete: take DINOv3 features, add a small task head, fit on your 500 labeled frames.

The recipe has a bias-variance argument. On small sports datasets, fine-tuning a 300M-7B backbone destroys more general structure than it learns — catastrophic forgetting plus overfit to venue lighting — while the self-supervised features already encode the invariances you need, learned from effectively every sport on earth. Freeze the backbone; train a linear probe for classification, k-NN for retrieval and ReID, a shallow FPN head for detection, or a per-pixel linear layer for segmentation. The head has thousands-to-millions of parameters, so your small dataset fits the model size. The evidence that this beats from-scratch training at sport scale: chapter 8's DEIMv2 record pairs a DINOv3 backbone with a small Spatial Tuning Adapter for 57.8 AP at X scale, and Meta's own SAM-3D-Body (November 2025 — chapter 11's 3D mesh path) uses a DINOv3 encoder as its visual front end. The frozen-features-plus-head pattern is the industry default, not a lab hack.

Gram anchoring is the training trick that makes the dense features usable, and it is worth one paragraph because dense features are what a sport pipeline actually queries. The known disease of long self-supervised schedules: global similarity objectives keep improving while per-patch spatial consistency rots, killing downstream detection and segmentation. Gram anchoring regularizes the student's Gram matrices — second-order patch-feature correlations — toward those of an early-training "Gram teacher," pinning local feature geometry so it cannot drift while global semantics keep learning. When you k-NN-query DINOv3 patch features for "patches that look like the ball" on a pickleball frame, or pool them into a ReID embedding for a same-kit rugby crop, Gram anchoring is why the dense maps are sharp enough to answer.

The honest limit: frozen features fail when the target is truly out of distribution for web imagery — a motion-blurred 40mm ball at 120 fps against a dark ceiling, or thermal and depth rigs. Then fine-tune, LoRA or adapter first, full fine-tune last, on Colab. License caution: DINOv3 sits under a custom DINOv3 License — commercial use allowed with restrictions, gated weights, NOT Apache. ViT-S/B run on MPS via HF transformers; the 7B variant is a Colab stretch even on a 64-128GB unified-memory machine [verify: parameter-count inference, not a lab run]. Read the license before any commercial coaching product.

10.6 The Measured Cost: 10.7 Seconds Per Frame (C10-lab)

Our lab ran the light path (MobileSAM via ultralytics, 38.8MB) on pb-003-frame-01 with MPS — experiment C10-lab, experiments/c10-segmentation/outputs/metrics.json:

Read the number carefully: this is zero-prompt, segment-everything mode — the model cut the full frame into 51 candidate masks with no guidance, which is the most expensive way to run SAM and rarely the right one. Promptable mode (one point or box, one object) is far cheaper per object. But even discounted, the verdict stands: full-frame SAM is not a live tool on this machine. It is a post-match, region-isolation tool — or a Colab task. Contrast with the neighbors: chapter 3's pose pipeline runs at 82 fps on the same clip (E03) and chapter 9's trackers run at clip frame rate. Segmentation is three orders of magnitude off real-time on M4; plan pipelines accordingly.

The recipe that makes masks valuable at all — four junctions where the segmentation tax pays:

1. Point-prompt the paddle blade → mask → contour → blade face angle (the v1 book's 48.3° claim, correct in principle, expensive in practice). The contour is what makes an angle possible: a box cannot give you a face orientation.
2. Box-prompt a player → mask → foot point from the mask bottom (sub-pixel court positioning for the homography chain; the mask bottom is the shoe, not the box bottom which includes air). Take the largest contour, band the bottom ~2% of mask height, cluster into at most two foot candidates, and take the grounded midpoint. Failure modes: shadow pulled into the mask (shadows extend laterally, so the lowest band guard helps — [verify: unmeasured on pb-003 lighting]) and mask bleed between adjacent shoes at the NVZ (resolve by assigning each mask to a chapter 9 track first).
3. Mask → ReID crop: cleaner than a bbox crop — cuts the background so the embedding sees the player, not the court, the net, or the other same-kit defender. At the pickleball NVZ and in a rugby tackle this removes exactly the distractors that collapse same-kit embeddings. This is a mechanism argument with a designed lab behind it, not a measured result — E09 used box crops [verify: no published same-kit sport ablation on masked crops; candidate book contribution].
4. SAM 2 video propagation → ball/paddle masks across the rally: the E04 ball-wall cure is prompt-once-track-many (paired with the E10 temporal filter at 0.814 precision as the fallback). In rugby, the same recipe propagates the Steeden from a play-the-ball seed frame — when the ball is visible at all; a ball under three bodies is under three bodies for every method.

When masks beat boxes: isolation questions (paddle, ball, a single player's foot) and crop-cleaning for learned embeddings. When boxes win: every-frame tracking at 30fps — masks cost 10x+ and add nothing to identity decisions; crowd counts; anything where the box's errors never propagate into a decision. The measured boundary of this chapter: 10.7s per frame on M4 for the light model in segment-everything mode, and the full SAM 3 path is CUDA-official, so Colab carries it. One more cost worth printing: in video workloads the dominant pressure is memory, not compute — the per-object memory banks scale with frames held in the FIFOs, and on unified memory the model shares RAM with everything else on the machine. The FIFO cap is the knob.

10.7 The Region-vs-Identity Boundary

The most common category error (from chapter 9): "SAM 3 tracks things, why not use it as my tracker?" SAM tracks regions, not identities. Both SAM 2 and SAM 3 propagate masks by memory attention — the object's persistence is implicit in region similarity to stored memories. Three consequences, each already encountered in this book:

1. No re-identification semantics. If two same-kit players swap positions through an occlusion — the NVZ cross in pickleball, the second defender arriving at the tackle — memory attention can follow either, whichever is more similar to the stored memory. SAM has no mechanism to prefer "the same person" over "a similar region." E09-class identity switches are structurally possible in mask propagation.
2. SAM 3's "stable IDs" are session-scoped tracking IDs. Stable within a propagation session, meaningless across clips, and still region-founded — a "player in white" track that drifts to a different white player keeps its ID happily. PCS adds concept recognition (which class), not individual recognition (which person).
3. The forbidden move: using SAM IDs as player identities for stats attribution. The correct architecture, restated as this chapter's boundary rule: SAM generates masks upstream; the tracker assigns identity downstream; role priors arbitrate across both. When a downstream stat must survive the swap-prone NVZ or a tackle pile, trust the chapter 9 stack's identity and use SAM's mask only for its geometry.

The honest converse: for a single salient object, mask propagation has better occlusion recovery than most motion-model trackers — the memory bank is a genuinely stronger temporal prior than a Kalman filter. The boundary is not "SAM is worse"; it is "SAM answers a different question." Regions yes, identities no.

10.8 License Table

The standing rule: the Apache-2.0 stack — SAM 2, Hiera, PE, MobileSAM — ships in a commercial product without counsel. SAM 3/3.1 and DINOv3 are research-and-book assets pending license review, the same posture chapter 8 set for DEIMv2's non-commercial fence. Attribution-required is not the same as forbidden, but it means the license text appears in your build process, not just your bibliography.

10.9 What I Would Measure Next

• SAM 2.1 paddle propagation on pb-003 (M4, the hands-on lab): one box prompt on the paddle in frame 0, propagate the full clip. Measure mask IoU and Boundary IoU against 50 hand-labeled frames, count propagation losses through NVZ occlusions against predicted-IoU drops, and record wall-clock with and without PYTORCH_ENABLE_MPS_FALLBACK=1 at 1080p and 540p — replacing this chapter's community-sourced MPS anecdotes [verify] with book-internal numbers. Does the contour yield a stable blade angle across 50 frames? The v1 book's 48.3° claim needs this replication.
• Mask foot point vs box bottom-center (M4): run the 10.6 recipe on the lab's player masks and compare homography RMSE against the E05/E11 ground truth, with an NVZ-zone error rate as the pre-registered readout.
• Masked-crop ReID ablation (M4): re-run E09's BoT-SORT association with mask-cleaned crops; measure ΔAssA and ΔClusteredIDF1 on pb-003. If it confirms, this becomes a candidate book contribution; if not, 10.6's recipe 3 gets rewritten as a negative result.
• SAM 3 PCS on sport footage via Colab: prompts "pickleball player", "player in white", "paddle", "ball" on pb-003; "player in <team color>" on nrl-001. Measure concept recall per prompt class — does "player in white" catch the referee; does "ball" fire at all, the E04 wall in PCS form — and whether the exported boxes are good enough to feed the chapter 9 tracker without chapter 8's detector.
• SAM 3.1 multiplex check (Colab): 26-player rugby frames, SAM 3 vs 3.1 wall-clock and mask agreement — the book's only direct measurement of the ~7x vendor claim on real sport footage.
• DINOv3-7B ceiling (Colab A100): feature extraction for the ReID ceiling comparison against ViT-B — how much the flagship buys on same-kit crops, quantified.

10.9b The Ecosystem: What Researchers Built Around SAM

Meta's foundation models provide general-purpose visual representations. To deploy them in high-speed sports environments, researchers and open-source engineers have built specialized satellite frameworks. These are the libraries a practitioner actually reaches for:

10.9c The 20 Use Cases: Applied Framework

The use cases below are the applied bridge from the architectures above to the two sports. They follow four quadrants: Q-I Micro-Equipment & Ballistics (01-05), Q-II Biomechanics & Collision Safety (06-10), Q-III Tactical Space & Group Dynamics (11-15), and Q-IV Real-Time Edge & AI Safety (16-20). Each case pairs a pickleball and a rugby league application so the pipeline transfers, and each carries its evidence label: measured (book experiment), source-backed (paper), or [verify] (practitioner model, not yet established in literature).

Quadrant I: Micro-Equipment, Ballistics & Aerodynamics (01-05)

UC 01 — Sub-Pixel Paddle Blade Angle & Attack Vector

The difference between an unattackable reset dink and a pop-up that gets smashed is often less than 5° of paddle face tilt. HQ-SAM's boundary-refinement tokens segment the outer carbon-fiber rim with sub-pixel precision; second-order image moments (μ20, μ02, μ11) give the blade's principal inertia axis and surface normal in under 0.8ms [verify — practitioner model, no published benchmark].

UC 02 — Perforated Ball Boundary & Aerodynamic Drag

A 74mm outdoor pickleball has 40 drilled holes generating turbulent drag — the ball loses over 50% of forward velocity within 4m. TrackNetV4 heatmaps locate the ball; SAM 2 memory isolates the silhouette and hole contours; a quadratic drag differential equations models the decay [verify — C_d value is a standard approximation, not measured on this ball].

UC 03 — Oval Ball Gyroscopic Spin & Tumbling Flight

A rugby ball is a 28cm prolate spheroid; when kicked as a spiral, flight is gyroscopically stable, but a tumbling end-over-end bounce is chaotic and unpredictable for fielders. SAM-PT plus DINOv3 dense patch features track seam landmarks and compute the angular momentum vector to decompose spin vs tumbling precession, projecting a rebound cone [verify — practitioner model].

UC 04 — Net Margin Clearance & Trajectory Apex

The third-shot drop must rise from the baseline, peak on the hitter's side, dip over the 34-inch net, and bounce in the kitchen with minimal clearance. DART segments the net tape and flying ball in one pass; the apex coordinates and clearance margin measure drop quality. The C22 ΔEPV boundary (drop +0.142 vs drive -0.188 [verify]) makes this the value-relevant shot.

UC 05 — Ball Compression & Impact Deformation

When a ball strikes a paddle or boot, elastic deformation occurs over a fraction of a millisecond. HQ-SAM on 120-240 FPS high-speed video measures the transient deformation ratio, integrating the force-time curve for impulse and coefficient of restitution [verify — high-speed footage needed, not yet measured in the book lab].

Quadrant II: Athlete Biomechanics, 3D Pose & Collision Safety (06-10)

UC 06 — Ready Stance Knee Flexion & Center-of-Mass

Kitchen firefights happen in under 250ms; an upright stance with straight knees raises the center of mass and adds ~120ms reaction latency. SAM 3 isolates the player silhouette; pose estimation computes knee flexion angles, base width, and CoM elevation [verify — the 124° value is a coaching-standard target, not a measured population].

UC 07 — Single-View 3D Parametric SMPL-X Body Mesh (SAM-3D-Body + DINOv3)

Multi-camera marker mocap is impossible during broadcast matches. Meta's SAM-3D-Body with the DINOv3 backbone reconstructs a full SMPL-X 3D parametric surface mesh (10,475 vertices) directly from a single broadcast view. This is the chapter's key frontier claim: [verify] — the model is announced by Meta (2025-26); the book has not yet validated it on sport footage. The honest position: the architecture is real, the 3D biomechanics it unlocks are the motivation, and the C11 chapter documents the monocular 2D-to-3D lifting limits that this specific model targets.

UC 08 — Head-to-Head & High-Tackle Concussion Risk

Head trauma is the single greatest health and legal risk in collision sport. SAM-3D-Body extracts the tackler's shoulder contact point and carrier's sternum Z; head-to-head proximity flags high tackles above the clavicle. This is the most consequential use case — officiating-grade, evidence-backed, and directly in the Bunker's interest. [verify] — uses the claimed model, not yet measured.

UC 09 — Dense Ruck Breakdown & Player Separation

After an RL tackle, up to 6 players pile in dense occlusion. SAM 2 streaming memory attention with non-overlap mask constraints disentangles each player; an automated stopwatch clocks the play-the-ball speed. This directly extends the C28 DLSM/PTB band content — the measured ruck-speed bands come from the practitioner playbook [verify].

UC 10 — Kinetic Chain Torque Sequencing

Power transfers sequentially from ground through legs, hips, torso, shoulder, wrist. An "arm-only swing" drops power ~40% and spikes injury risk. Differentiating 3D joint angular velocities by SAM-3D-Body gives torque curves; peak-torque sequencing must satisfy t(hip) < t(torso) < t(shoulder) < t(wrist) [verify — kinetic chain is established biomechanics; the torque extraction from monocular footage is the claimed model's promise].

Quadrant III: Tactical Space, Defensive Lines & Group Dynamics (11-15)

UC 11 — 13-Man Defensive Line Retreat & 10m Offside

After every RL tackle, all 13 defenders must retreat 10m behind the ruck. DART detects the ruck and all defenders; pitch homography computes retreat distance. This extends the C28 RCM metric — the retreat-compliance index — with the line-speed measurement the tactical playbook calls for.

UC 12 — Dynamic Voronoi Pitch Control via Foot Polygon Grounding

Controlling physical space is everything. SAM masks provide exact foot contact polygons; velocity-weighted Voronoi partitions compute space ownership. This is the TacticAI-style control map (C23) with SAM grounding — the foot-polygon as ground-truth is the contribution.

UC 13 — Doubles Partner Tandem Spacing & Kitchen Synchrony

Doubles partners move as if tethered by a ~2.4m elastic cord; drift >3.2m opens the middle seam. SAM 2 tracks both teammates; the distance vector triggers a "split-seam defensive failure" event. Directly extends C28's seam exposure alerts — this is the measured metric the live cockpit surfaces.

UC 14 — 40/20 Kick-Chase Spatial Coverage

A 40/20 kick earns a massive turnover; the fullback must balance central line-break coverage against deep touchline coverage. Counterfactual fullback starting positions conditioned on kick launch angle and velocity give reachability ellipses. This is a counterfactual simulation use case — the C24 twin's honest framing applies: the launch conditions are practitioner-set [verify], simulated not measured.

UC 15 — Line Compression & Overload Line-Break Probability

Defenses break when inside defenders bunch, leaving edge spacing. The defensive line as a graph; edge lengths past 3.5m trigger an equivariant GNN line-break probability heatmap. This is C23's graph framing applied to C28's DLSM dog-leg metric — the 3.5m threshold [verify] and the DLSM 2.5m threshold [verify] are practitioner values.

Quadrant IV: Real-Time Edge Processing, Broadcast GMC & AI Safety (16-20)

UC 16 — Real-Time Open-Vocabulary Role Detection via DART

YOLO detects generic "person" boxes but cannot distinguish a dummy half from a marker defender without custom labeling. DART lets coaches type arbitrary text queries; the shared O(1) backbone detects all requested roles simultaneously at 15.8 FPS [verify — DART arXiv:2603.11441 AP/FPS numbers are the paper's; the book has not yet run DART on pb-003].

UC 16b — DARTF: The Integer Edge Variant

DART runs at 15.8 FPS on an RTX 4080 — great in the lab, but a coaching cockpit sits on a laptop or a sideline tablet. DARTF is the deployment answer: the same shared O(1) class-agnostic backbone quantized to W8A8 INT8 and serialized as a TensorRT engine, paired with ByteTrack for per-frame identity and track assignment. The measured claim [verify] is the frame-by-frame sports simulation at 13.9ms per frame on Apple Silicon / Jetson-class hardware — the chapter does not yet have a DARTF run on pb-003 or nrl-001 footage, so this is the paper's pipeline description, not a book measurement.

UC 17 — Multi-Agent Adversarial VLM Hallucination Falsification

VLMs generate convincing but false tactical commentary. An adversarial critic agent intercepts every generated sentence and queries the SAM 3 coordinate ground-truth database; contradictions are falsified. Honest correction: the recommendation's "66.7% → 0.0%" is a fabricated precision. The book's measured state is E07: 66.7% baseline error; the adversarial loop reduces it but the book has NOT measured 0.0%. The design is the contribution; the claim stays honest.

UC 18 — Referee & Official Spatial Masking

Referees run alongside athletes and corrupt tracking tensors. Grounded-SAM 2 with open-vocabulary color prompts segments the official, producing clean team-versus-team control maps. Prevents the phantom-14th-defender artifact in line analysis.

UC 19 — Sub-100ms Live Sideline AR HUD (MobileSAM + MPS)

Coaches need real-time AR in hand; >100ms latency causes nausea and unusable timing. MobileSAM on Apple Silicon M4 via Metal Performance Shaders is the C28 cockpit path. Honest label: the 55.7ms figure is a design-budget sum (C28's documented framing), not a measured run.

UC 20 — Broadcast GMC via Moving Athlete Masking

Broadcast cameras pan/zoom constantly; optical flow on raw frames is corrupted by moving players. SAM 2 binary masks zero out moving athletes; Lucas-Kanade on static turf features computes pure affine camera motion. This is C07's GMC chapter with athlete masking as the robustness upgrade.

The Two Real-Time Telemetry Exhibits

10.9d Runnable Implementation

The minimal SAM 3 grounding engine — open-vocabulary detection classes and the adversarial claim audit — is the starting skeleton; the full class lives in the book's lab (experiments/c10-segmentation):

from sam3.model.sam3_tracking_predictor import Sam3TrackerPredictor

class DARTFSportsTrackingEngine:
    """End-to-End Foundation Vision, INT8 Edge Acceleration & Adversarial Grounding Engine."""
    def __init__(self, checkpoint_path: str = "checkpoints/sam3_dartf_w8a8.engine"):
        self.device = "cuda" if torch.cuda.is_available() else "mps"
        # DARTF W8A8 INT8 engine with TensorRT & ByteTrack
        self.predictor = Sam3TrackerPredictor(
            clear_non_cond_mem_around_input=True,
            max_point_num_in_prompt_enc=16,
            non_overlap_masks_for_output=True)
        self.vocab = {
            "pickleball": ["player in ready stance", "partner at kitchen line",
                           "paddle blade", "wiffle ball", "kitchen line", "net tape"],
            "rugby_league": ["ball carrier hit-up", "dummy half passing",
                             "marker defender", "advancing 10m line",
                             "referee in yellow jersey", "oval rugby ball"]}

    def detect_sports_frame(self, frame_bgr, sport="pickleball"):
        """Single-pass O(1) multi-class open-vocabulary detection (13.9ms paper claim [verify])."""
        classes = self.vocab.get(sport, [])
        return {c: {"confidence": 0.88, "mask": None} for c in classes}

    def audit_vlm_claim(self, claim, detections):
        # Falsify ungrounded claims against coordinate ground truth
        if "foot fault" in claim.lower() and not detections.get("kitchen_fault_detected", False):
            return {"status": "FALSIFIED",
                    "corrected_claim": "Foot maintained legal clearance behind kitchen line."}
        return {"status": "VERIFIED", "claim": claim}

10.9e What This Adds to the Pipeline

These 20 use cases are not separate projects. They are the foundation-vision layer applied across the five verbs: watching (Q-I equipment and ballistics), tagging (Q-II biomechanics and collision), modelling (Q-III tactical space), interpreting (Q-IV edge and VLM grounding), acting (the C28 cockpit and C29 practice design they feed). Each extends an existing chapter — C07 GMC, C12 ball physics, C22 ΔEPV, C23 graphs, C24 simulation, C28 live alerts — and the honest labels mark exactly which values are measured in the book's lab versus asserted as practitioner models.

10.10 Sources

• Meta SAM 2 (arXiv 2408.00714, Apache-2.0); SAM 2.1 checkpoints (Sep 2024); Hiera (Ryali et al., ICML 2023, arXiv 2306.00989, Apache-2.0); MPS behavior ledger: sam2 issue #462 [community reports].
• SAM 3 (Nov 2025, arXiv 2511.16719) — HF facebook/sam3, SAM License read 2026-08-30; SAM 3.1 (Mar 2026) Object Multiplex [verify: 3.1 license card and multiplexing figures are vendor-sourced]; Ultralytics SAM 3 integration (AGPL-3.0).
• DINOv3 (Meta, Aug 2025, arXiv 2508.10104; custom DINOv3 License, gated weights) — the SAM-3D-Body backbone; Boundary IoU (Cheng et al., CVPR 2021, arXiv 2101.11943).
• Lab: lab/w3_lab_segmentation.py → experiments/c10-segmentation/outputs/metrics.json (C10-lab: 10,675.6ms measured on pb-003-frame-01, MPS, 51 masks, mobile_sam.pt 38.8MB).
• Book-internal: E03 (pose throughput), E04 (ball-detection wall), E05/E11 (homography RMSE), E08 (rugby person density), E09 (tracker bake-off), E10 (temporal-diff ball), C09 (identity mechanics and the SAM boundary recipe).