AS '26
All Chapters

Tagging · SECTION 15

Chapter 15 — Audio & Multimodal Cues: The Free Sensor

Paddle pops, whistles, and crowd roar: why the audio track is your cheapest event detector

Reading time

29 min

15 Chapter 15 — Audio & Multimodal Cues: The Free Sensor

Paddle pops, whistles, and crowd roar: why the audio track is your cheapest event detector

15.1 Audio Proposes, Vision Confirms

Every match recorded over a broadcast feed or a court camera already carries an audio track — and that track contains timestamped physics: a paddle hit, a ball bounce, a referee's whistle, a tackle's thud. Vision struggles to time these precisely (the ball is 8px and hides behind hips); audio times events to millisecond precision at zero vision cost. Timing density is the argument: a 44.1kHz track stamps a contact to 0.023ms — roughly 1,500× denser temporal sampling than 30fps video frames, which quantize any event to ±16.7ms at best. No occlusion, no homography, no calibration, no model. The 2026 fusion rule this chapter teaches: audio proposes an event timestamp, vision confirms the event's content.

The cheap-sensor economics deserve stating plainly. A MEMS microphone element costs about a dollar in quantity, and the microphone on the phone or camera you already own costs nothing extra at all. That one-dollar sensor catches three event classes that cost the vision stack dearly: the pickleball ball-wall bounce that E04's detector wall (1,082 candidate detections at median confidence 0.11, measured) could never time cleanly, the rugby tackle impact that is visible only as a pile of bodies, and the referee whistle that is not visible at all. An onset detector that harvests these events is roughly twenty lines of librosa running on CPU. Compare the vision path to the same timestamp: detector (E04's noise floor) → tracker (E09's ByteTrack/BoT-SORT machinery) → pose (E03, 16.46 kps mean) → contact inference. Audio skips the entire stack for the when.

The evidence supplies the motivation: the pb-003 manual contact review found 0 of 6 vision-confirmed contacts in the still-frame set. Vision alone cannot time a contact on this fixture; audio can. Both book clips carry usable sound — the nrl-001 broadcast audio and the pb-003 source audio are ffprobe-verified stereo AAC 44.1kHz — so everything in this chapter runs on media the reader already has. All lab numbers below are measured on our own clips; external claims are marked, and practitioner-model numbers carry [verify].

15.2 The Signals: A Two-Sport Taxonomy

Audio events divide into three acoustic families, and both of our sports populate all three. Impulsive transients are broadband spikes a few milliseconds long — the pickleball paddle pop, the ball-court bounce thud, the rugby tackle impact on a field-side mic. Sustained narrowband tones are the referee whistle's signature: a stable, periodic tone holding energy in roughly the 2–4kHz band for 0.3–1s [verify per whistle model]. Slow broadband beds are the crowd: spectrally stable, varying on a 0.1Hz scale, correlated with moment salience but never an event trigger. The families matter because each wants a different detector: transients want spectral flux, tones want narrowband-ratio and flatness features, beds want slow RMS envelopes.

Audio Event Acoustic Family What It Means Sport
Paddle/ball contact (pop) Impulsive transient Shot event timestamp — zero vision cost; inter-onset interval = rally tempo Pickleball, tennis, badminton
Ball bounce (thud) Impulsive transient, low-weighted Separates rally state; bounce-side prior for the C10 event log Pickleball
Net contact (twang) Impulsive transient, mid-band ring Let cord / net-cord anomaly flag Pickleball
Referee whistle Sustained narrowband tone Decision moment (held, knock-on, six-again, 10m penalty); hard segment boundary Rugby League
Tackle impact Impulsive transient Collision timestamp on field mics; buried under commentary in broadcast mixes (gap, §15.8) Rugby League
Crowd roar Slow broadband bed Pressure proxy; moment saliency for review prioritization — correlational, never causal Both
Audio cue taxonomy tree with two branches: pickleball (pop, thud, twang) and rugby league (whistle, roar, tackle), each leaf with a line icon.
Figure 15.2: The two-sport audio cue taxonomy. Every leaf is detectable with the same three-family machinery; only the confounds differ per sport.

The division of labor is asymmetric on purpose. Audio owns when; it can never say who hit the ball or where it landed — a single microphone has no geometry. Vision owns what, who, and where but pays frame quantization, occlusion, and detector noise for the privilege. That split, not any model choice, is the chapter's spine.

15.2b The Two-Sound Problem: Pop vs Thud

Pickleball's core audio problem is that the two most important rally events — paddle contact and ball bounce — are both impulsive transients from the same ball, often inside the same second. Distinguishing them is a spectral problem, and it is tractable because the physics differ. A paddle pop is a hard broadband strike: community-noise measurements of pickleball consistently describe contact as a short impulse with energy prominent around 1–2kHz [verify exact band on pb-003 — our own lab pass is the §15.10 plan], produced by polymer-on-polymer at arm's length from the mic. A bounce thud is the same ball against a court surface: lower-weighted energy, softer attack, shorter high-frequency content. On a spectrogram the pop is a tall stripe; the thud is a low blob. Per-onset features — peak flux, spectral centroid, decay time — separate them without any learning, and the same features yield a free drive-vs-dink prior: a drive's pop is louder with a sharper attack, a dink tap is the same spectral shape at lower amplitude into a quieter local floor. That prior disambiguates; it never labels a shot alone (that is vision plus court position, C10).

Rugby league has the same structural problem one family up: the whistle (narrowband tone) must be separated from the mic'd referee's voice, which shares harmonic content with the whistle band. The discriminator is stability — the whistle is amplitude- and pitch-stable where speech is modulated, so spectral flatness over time separates them. And one honest gap belongs here: the tackle impact, the transient fans care about, exists on field-side microphones but is sub-mixed beneath commentary and crowd in broadcast audio [INFERENCE] — treat it as a target for venue mics, not a broadcast feature.

Two spectrogram panels: paddle pop shown as a tall broadband vertical stripe; ball bounce shown as a low-frequency blob labeled thud.
Figure 15.3: The two-sound problem on a spectrogram. Pop and thud share the ball but not the spectrum — peak flux, centroid, and decay time split them (concept diagram; pb-003 measured bands pending §15.10).

15.3 Methods: The Three Escalation Lanes

Lane 1 — librosa spectral-flux onsets (the 80% solution, ✅ M4): an onset is a frame-to-frame spectral magnitude change, so a transient spikes the flux regardless of the steady-state noise floor that kills naive RMS thresholds; backtrack=True snaps each detection to the local energy rise, recovering the true contact instant rather than detector lag. Combine onset times with per-window spectral centroid to classify brightness: a whistle is bright and tonal (4kHz+ peak), crowd is low rumble (<1.5kHz), commentary sits between. This is what our lab ran — 107 onsets in 20s of nrl-001 audio (measured, §15.4) — and it is the entire pickleball-contact solution until SNR kills it. Its failures are visible on a spectrogram, which is why you start here.

Lane 2 — MFCC + small CNN (whistle-vs-confounds classifier, literature-verified): MFCC frames with delta-MFCC for transient dynamics — drop MFCC-0, which carries loudness rather than timbre, so the model stays invariant to broadcast mix compression — fed as a 2D coefficient matrix to a small CNN. The recipe is on record for exactly our problem: CNN whistle detection distinguishing referee whistles from supporter whistles and excited commentary (MDPI Applied Sciences 2022, 12(23):12151, CC-BY). The known trap is weak labels: training clips mark "whistle somewhere in here," not the onset — use weakly-supervised framing or hand-label onsets for your own few hundred clips. Train on a Colab free tier, export ONNX, infer on the M4.

Lane 3 — pretrained audio foundation models: PANNs CNN14 (MIT, ✅ CPU-real-time, AudioSet "Whistle"/"Crowd" classes zero-shot), YAMNet (Apache-2.0, ⚠️ TF-on-macOS friction), AST (MIT, ⚠️ heavier attention over spectrogram patches), and the 2026 audio-first multimodal models — Phi-4-multimodal (MIT, Feb 2025, 5.6B, runs quantized locally) and Qwen3-Omni (Apache-2.0, Sep 2025; successor Qwen3.5-Omni Mar 2026, license [verify]) — which make audio a first-class perception channel and can name the referee's call as semantics, not just detect a transient. Enforce the division of labor: these are windowed semantic layers (~0.5–1s decisions), not millisecond timers. Lane 1 stamps the event; lane 3 narrates it.

Audio event detector pipeline: microphone to FFT to spectral-flux onset curve to onset timestamp list to classifier branching to whistle, pop, crowd.
Figure 15.4: The lane-1 detector pipeline. Everything left of CLASSIFY is sample-accurate and training-free; escalation to lanes 2–3 replaces only the last box.

15.4 What We Measured (C15 Lab)

We extracted 20s of the nrl-001 broadcast audio (ffmpeg -vn -ac 1 -ar 22050, mono WAV) and ran the lane-1 onset + centroid stack (experiments/c15-audio/outputs/metrics.json, measured):

Measurement Value
Onsets detected (20s) 107 (well-timed event candidates; 0.19s median spacing) — measured
Tempo estimate 103 BPM (crowd rhythm, not a signal) — measured
Window classes voice/commentary (peaks at 2430–2497Hz) vs transient_bright (4103Hz = whistle-class) — measured

The honest reading: onset detection finds transients — but separating WHISTLE from CROWD needs either a trained classifier or a large spectral contrast. This baseline proves the raw signal is rich enough for audio-proposes-vision-confirms fusion; it does not claim a working whistle detector. The 4103Hz transient_bright window is a whistle-candidate, and the broadcast mix's compression means spectral shape (narrowband ratio, flatness) is the trustworthy feature, never raw energy. On the pickleball arm, the pb-003 source track (verified stereo AAC 44.1kHz) awaits its own pop-detection pass — the six contacts vision could not confirm are exactly the rows audio should close (§15.10).

Three stacked spectrogram strips for rugby league: whistle as a sustained narrowband line near 3kHz, crowd roar as a swelling low-frequency band, tackle as a single short broadband stripe.
Figure 15.5: Rugby league's three audio events by family. The whistle's tonal stability is the detector's friend; the tackle stripe is a field-mic event, not a broadcast one (concept diagram).

15.5 The Multimodal Fusion Rule

  1. Audio onset at t=1.53s (a bright transient — a real onset from the measured list above).
  2. Look up the video frames at t ± 50ms: is there a ball-position discontinuity (the E10 temporal-diff cue, 0.814 precision on record) or a pose-velocity spike at the contact limb (E03 stream)?
  3. Record the event only when audio AND vision agree.

The asymmetry is load-bearing. Audio is high-precision timing with low semantics; vision is high semantics with low-precision timing (frame quantization, E04's noise floor, occlusion). Running the fusion in reverse — vision proposes, audio confirms — would force the ball detector past its demonstrated wall before the one-dollar sensor even gets a vote. The proposed direction spends vision only where it is strong: a ±50ms crop around a known instant, with tracking context (C09) already warm. Fusion output carries a two-bit confidence state with an explicit event-log policy: audio+vision is a full-confidence event feeding shot attribution (C10); audio-only stays a candidate — and is not a failure, because E04 teaches the ball is often invisible, so absent visual confirmation is weak negative evidence; vision-only is suspicious (missed pop or visual false positive) and flagged for review; none marks the silence between rallies, itself useful for rest-period stats.

Two fusion refinements matter per sport. In rugby, whistle onsets are hard segment boundaries: the match becomes a list of (previous whistle → this whistle) windows, and every downstream vision job — tackle detection, PTB speed, C09 re-tracking — is scoped to a window instead of 80 minutes of PTZ video. That is the single largest compute saving in this chapter. In both sports, the crowd channel modulates but never triggers: raise review priority for events inside a swell, annotate the log with a pressure flag, and expect a whistle within seconds of a sustained roar at goal-line defense — with the correlational-not-causal wording held in every claim (§15.8).

Audio-vision fusion: onsets into t±50ms junction, vision frames into same, four confidence states. Recreate: white background, burnt-orange banner.
Figure 15.1: Audio Proposes, Vision Confirms. Two lanes meet at a ±50ms junction and produce one of four confidence states. Only "both" records an event.
Four confidence state cards: A+V solid border with waveform and eye, A ONLY dashed with waveform, V ONLY with flag, NEITHER empty dotted.
Figure 15.6: The two-bit confidence states and their event-log policies. The audio-only card survives because E04 says the ball is often invisible.

15.6 Sync: The Practical Discipline

The fusion rule lives or dies at the timestamp boundary. Target ±50ms — comfortably inside broadcast A/V-tolerance practice (ITU-R BT.1359 [verify exact ms values]) and fine enough to look up the right frame for any rally event. Sync is a measurement, not an assumption: C04's clock math applies to the audio sample clock as much as the video frame clock, and one clap at each end of the session plus one cross-correlation check per device pair retires the entire risk class for minutes of effort.

Source Drift Fix
Phone clock 10-20 ppm ≈ 1ms/min (30min = 30-60ms ≈ 2 frames @30fps) Loud clap at start and end; compute drift rate, stretch one timeline; re-clap every 15-20min
Multi-camera / separate recorder different clocks, arbitrary offset Genlock / timecode (pro); clap or flash (amateur); GCC-PHAT cross-correlation (Knapp & Carter 1976) recovers sub-millisecond offset from ambient audio alone — pure NumPy, ✅ M4
Broadcast (nrl-001) already muxed Verify once: a visible whistle arm-signal or kick impact should land within a frame or two of its onset

On the 30s book clips, drift is negligible (<1ms); the protocol matters the day you record a full session. GCC-PHAT is also the load-bearing technique for any multi-mic rig (§15.7) and for marrying a courtside recorder to camera audio in either sport.

15.7 The Industry Pattern: Zenniz's 30-Mic Array

Verified: Zenniz (tennis) combines 30 audio sensors + 4 cameras for ball tracking and line calling, with 25+ countries deployed — the audio-first outlier in a smart-court market that is otherwise camera-only, and evidence the pattern survives commercial reality. The architecture lesson: distributed audio + multi-camera + central fusion. The physics is time-difference-of-arrival (TDoA) multilateration: each mic stamps the impact, and arrival-time differences across the array localize it in 3D. The numbers make it concrete — 1ms = 34.3cm (c = 343 m/s); one sample at 48kHz = 7mm of sound path — so a sample-synchronized sensor ring has physically comfortable headroom for sub-decimeter impact localization [INFERENCE from geometry; Zenniz internals are proprietary]. The cameras then never hunt a 200km/h ball from scratch: audio says when and roughly where the bounce happened; vision confirms IN/OUT at the line. Audio-proposes-vision-confirms is a shipping architecture, not our invention.

Overhead pickleball court with four corner microphones, an impact point near the kitchen line, and intersecting TDoA hyperbola arcs.
Figure 15.7: TDoA gating at hobbyist scale: corner mics, GCC-PHAT-synced, reject any pop whose arrival-time difference falls outside the known court polygon — the pickleball adjacent-court-bleed fix (concept diagram).

What the book's rig copies at a hundredth of the price: 2–4 USB or phone mics at the court corners, GCC-PHAT-synchronized (§15.6), TDoA-gated to the known court polygon. That gate is also the answer to pickleball's signature confound — an adjacent court's pop is the same event class, spectrally indistinguishable, and only geometry (or vision fusion) can reject it [proposed lab extension, not yet run].

15.8 The Limits Table

Limit Mechanism Consequence / Fix
HVAC hum 50/60Hz + harmonics raise the noise floor under everything Stationary low-band noise; flux-invisible to onset detection; a high-pass filter is the cheap fix. Indoor pickleball: high severity
Outdoor wind Broadband gusts and mic buffeting bury one-shot pops Wind screens or indoor capture; the pop lives above ~1kHz, so a 200Hz high-pass helps. Outdoor pickleball: high
Adjacent-court bleed Neighbor paddles are the same event class in the same band — spectral features cannot reject them The pickleball confound — TDoA gating or per-court directional mics; amplitude gating is fragile under phone AGC
Whistle ≈ referee verbal Mic'd referee speech shares harmonics with the whistle band Spectral flatness + duration separate them; a lane-2 CNN handles supporter whistles and commentary squeals. Rugby: moderate
Broadcast compression Mix compressors flatten amplitude cues; loudness features lie Onset detection still works; use spectral shape (narrowband ratio, flatness), never raw energy. Rugby broadcast: structural
Phone AGC Automatic gain rescales amplitude between events A loud crowd can mask a pop and corrupts drive-vs-dink amplitude priors; disable AGC where the app allows, else use relative/spectral features only
Tackle impact sub-mixed Collision thuds sit under commentary and crowd in broadcast mixes [INFERENCE] Field-mic feature, not broadcast; flag as a gap, not a capability
Crowd as proxy Correlational only — Nevill/Balmer/Williams 2002 found ~15.5% fewer fouls against the home team with crowd noise; COVID ghost games shrank but did not erase home advantage Crowd pressure on referees is real; crowd→player-performance is equivocal. Context channel, never causal wording
Privacy Venue audio captures bystander speech Broadcast commentary contains voices — consent and C31 rights apply to re-publication; default to storing events, not raw audio

15.9 The Audio-Fusion Recipe (Copy This)

  1. Extract: ffmpeg -vn -ac 1 -ar 22050 WAV (the C15 lab's exact command).
  2. Onset-detect: librosa spectral flux with backtrack=True; stamp times; threshold on brightness (spectral centroid per window).
  3. Classify by event: whistle (2–4kHz narrowband, sustained, low flatness) vs pop (broadband transient, ~1–2kHz-weighted) vs thud (low-weighted blob) vs crowd (slow broadband bed).
  4. Sync check: loud clap at recording start and end; measure the A/V offset and drift rate; correct (±50ms target).
  5. Fuse: at each candidate τ, look at the video frames at τ ± 50ms — vision confirms the event, audio timed it.
  6. Record only when audio AND vision agree; keep audio-only as a candidate; flag vision-only for review (the two-bit confidence state).
  7. Escalate to an audio foundation model (Phi-4-multimodal local, Qwen3-Omni on Colab) when the event taxonomy needs semantics — e.g. which call the referee made.

15.10 What I Would Measure Next

  • Paddle-pop detection on pb-003's source audio: does each onset align with a contact frame (the E06 state machine input), and does the pop-vs-thud split reproduce on our own footage? Acceptance: every hand-audible pop matched within ±50ms; onset count vs manual rally count.
  • Whistle vs crowd classifier on the nrl-001 audio (a small CNN on MFCCs — the honest few-hundred-clip labeling task), with PANNs' zero-shot "Whistle" head as the baseline to beat.
  • Multi-mic time-of-arrival: 2–4 phone mics on a pickleball court — can GCC-PHAT-synced TDoA localize the bounce side and reject adjacent-court pops?
  • Referee semantics: a Qwen3-Omni pass over nrl-001 whistle windows — "did the whistle blow, and what did the referee say?" — the decision-content layer above the decision-timestamp layer.

15.10b The Audio Toolkit: What to Reach For

The use cases below run on a small, boring toolkit. The rule is the §15.3 escalation ladder in table form: start free and training-free, buy a classifier only for the whistle-vs-confounds problem, and reserve the foundation models for semantics. Every row lists the sport job it actually does.

Tool License / M4 Role in the Stack Pickleball Use Rugby League Use
librosa (onset_detect, spectral_centroid, RMS) ISC / ✅ Lane 1: sample-accurate onsets, spectral-flux transient detection, centroid classification Pop/thud detection, rally segmentation, drive-vs-dink prior Whistle-band candidate extraction, crowd-envelope channel
GCC-PHAT (Knapp & Carter 1976, pure NumPy) n/a / ✅ Sync and TDoA: sub-millisecond offset recovery between devices, multi-mic gating Corner-mic rig sync; adjacent-court pop rejection Camera-audio ↔ field-recorder alignment; sideline array
MFCC + small CNN (own code; recipe source-backed: MDPI Appl. Sci. 2022, 12(23):12151, CC-BY) MIT/Apache parts / ✅ infer, Colab train Lane 2: whistle-vs-confounds classifier (drop MFCC-0 for loudness invariance) Pop vs hand-clap vs adjacent-court classification Referee whistle vs supporter whistles and commentary squeals
PANNs CNN14 (arXiv:1912.10211) MIT / ✅ CPU-real-time Lane 3 baseline: AudioSet 527-class heads — "Whistle", "Crowd", "Applause" zero-shot Zero-shot pop/bounce class prior for labeling Zero-shot whistle baseline the lane-2 CNN must beat
YAMNet Apache-2.0 / ⚠️ TF-on-macOS friction [verify ONNX] Lane 3 edge baseline (MobileNet-depthwise AudioSet classifier) On-device tagging reference On-device tagging reference
AST (arXiv:2104.01778) MIT / ⚠️ heavier Lane 3 attention over spectrogram patches — stronger, costlier embeddings Shot-type embedding research direction Broadcast-scene embedding research direction
Phi-4-multimodal (5.6B, Feb 2025) MIT / ⚠️ quantized local, not real-time Zero-shot audio QA: "did a whistle blow here, and what was said?" Rally narration over audio windows Referee-verbal reading fallback (local)
Qwen3-Omni (30B-A3B, Sep 2025; successor Qwen3.5-Omni Mar 2026, license [verify]) Apache-2.0 / ⛔ full, Colab Joint audio+video semantics — one model that watches and listens Zero-shot rally QA on fused clips Whistle-window referee semantics: the call, not just the timestamp

15.10c The 10 Use Cases: Applied Framework

Everything above is machinery. This section is where it earns court time: ten concrete jobs, grouped into four categories — pickleball contact events (UC 01–02), rugby league decisions and impacts (UC 03–05), the fusion layer (UC 06–08), and rigging with honest limits (UC 09–10). Each use case states the practical problem, the exact mechanism and math, a figure, the pickleball and rugby league applications, and who collects the payoff. The starting skeleton for every contact-event case is the same dozen lines:

import librosa, numpy as np

y, sr = librosa.load("match.wav", sr=22050, mono=True)   # the §15.9 extract
onsets = librosa.onset.onset_detect(y=y, sr=sr, backtrack=True,
                                    units="time")                     # sample-accurate t
S = np.abs(librosa.stft(y))
cent = librosa.feature.spectral_centroid(S=S, sr=sr)[0]  # pop vs thud vs tone
for t in onsets:                                          # the fusion contract
    window = video.frames(t - 0.050, t + 0.050)           # ±50ms, §15.6 budget
    emit_event(t, confidence=two_bit(onset=t, vision=window))

Category I: Pickleball Contact Events (UC 01–02)

UC 01 — Paddle-Pop Contact Timing (the Vision Problem Audio Already Solved)

The practical problem: the pb-003 manual contact review confirmed 0 of 6 contacts from still frames (measured) — vision cannot time a kitchen-line contact on this fixture, full stop. The mechanism is lane 1: spectral flux measures frame-to-frame magnitude change, S(n) = Σk max(0, |X(n,k)| − |X(n−1,k)|), spiking on the pop regardless of the venue's noise floor; backtrack=True snaps the detection to the local energy rise, recovering the true contact instant. The math that matters is timing density: 44.1kHz stamps the event to 0.023ms against 30fps video's ±16.7ms quantization — roughly 1,500× denser, with no occlusion, homography, or model in the path. The pop band itself (~1–2kHz prominence) is community-noise literature until §15.10 measures it on pb-003 [verify].

Annotated spectrogram of a pickleball paddle pop: one tall broadband vertical stripe labeled POP with an orange onset marker, paddle-and-ball inset at the kitchen line.
Figure 15.8: UC 01 — The Paddle Pop. A broadband stripe, backtracked to its energy rise, at 0.023ms stamp density. Pickleball: contact timestamps, rally segmentation, serve detection from the first onset after the longest gap. Rugby league: boot-ball contact on kickoffs and restarts from field-side mics [verify broadcast audibility].

Payoff: officiating and coaching — contact timestamps enter the C10 event log at zero vision cost, converting an unsolved detection problem into solved signal processing.

UC 02 — Ball-Bounce Detection (the Second Transient)

The practical problem: the two most important rally events — paddle contact and ball bounce — are the same ball, same second, same impulsive family; a pipeline that cannot split them cannot score the two-bounce rule or localize a kitchen-fault review. The mechanism is spectral, and tractable because the physics differ: the bounce is lower-weighted energy with a softer attack and less high-frequency content. The discriminating features per onset are peak flux, spectral centroid C(n) = Σ f·|X| / Σ |X|, and decay time; on a spectrogram the pop is a tall stripe and the thud a low blob. Both bands await the pb-003 lab pass before shipping unhedged [verify].

Two spectrogram panels side by side: a tall broadband POP stripe and a low soft THUD blob, with an orange centroid bracket between them and a bouncing wiffle ball inset.
Figure 15.9: UC 02 — Pop vs Thud. Same ball, different spectrum: centroid height and decay split the contact from the bounce (concept diagram; measured bands pending §15.10). Pickleball: bounce-side prior for the two-bounce rule and C10 event log. Rugby league: grubber and bomb bounce instants on field mics, anchoring the C12 bounce model [verify on broadcast].

Payoff: rally state machines — the C19 eventing layer gets bounce boundaries without asking ball tracking to survive the E04 wall.

Category II: Rugby League Decisions & Impacts (UC 03–05)

UC 03 — The Referee Whistle: A Decision Clock for 80 Minutes

The practical problem: an 80-minute broadcast is one long PTZ video, and every downstream vision job wants stoppage boundaries it currently has to find by pixels. The whistle is the boundary — every knock-on, penalty, six-again, and half-time is whistle-anchored, and NRL referees are mic'd for broadcast so the tone sits hot in the mix. The lane-1 mechanism: narrowband energy ratio (2–4kHz band energy over total [verify per whistle model]), spectral flatness SF = exp(mean(log|X|)) / mean(|X|) — a whistle is tonal, so SF is low where crowd roar is noise-like and high — plus duration 0.3–1s. The lane-2 escalation is the source-backed recipe (MDPI Applied Sciences 2022, 12(23):12151, CC-BY): MFCC + Δ-MFCC with MFCC-0 dropped for loudness invariance, into a small CNN that rejects supporter whistles and commentary harmonics. Hand-label onsets; the literature's known trap is weak clip-level labels.

Spectrogram of a referee whistle: a sustained narrowband horizontal line labeled WHISTLE above a fuzzy crowd band, with a referee arm-signal inset.
Figure 15.10: UC 03 — The Decision Clock. A stable tonal line at ~3kHz over a noise-like crowd bed; flatness and duration are the features, loudness is the lie. Rugby league: hard segment boundaries — every vision job scopes to (whistle → whistle) windows instead of 80 minutes. Pickleball: officiated tournament stoppages and let calls get the same anchor.

Payoff: the largest compute saving in this chapter — whistle-window scoping cuts the vision workload for C09 re-tracking, tackle detection, and PTB speed to the minutes that matter.

UC 04 — Crowd Swell: The Energy State, Not an Event

The practical problem: review prioritization and highlight selection need a salience signal, and the venue is already shouting it. The mechanism is the third acoustic family: a slow broadband bed whose RMS envelope, low-passed at the 0.1Hz scale, swells with moment salience — line breaks, tries, goal-line stands, firefight endings. The math is deliberately boring: E(t) = LPF(RMS), flag a swell when E crosses a venue-relative percentile. The honesty is the feature: crowd audio modulates, never triggers. The evidence supports crowd→referee bias (Nevill, Balmer & Williams 2002: ~15.5% fewer fouls against the home team with crowd noise, source-backed; COVID ghost games shrank but did not erase home advantage — Reade & Singleton) while crowd→player-performance remains equivocal. Causal wording is banned in every claim.

Spectrogram of crowd noise: a slowly swelling low-frequency broadband bed labeled CROWD with an orange 0.1Hz envelope curve and a grandstand inset.
Figure 15.11: UC 04 — The Pressure Proxy. A slow envelope over a broadband bed; the caption rule is "never a trigger." Rugby league: a sustained roar at goal-line defense predicts a whistle within seconds and flags the window for review. Pickleball: small-venue surges mark firefight endings and upsets for recap reels.

Payoff: editorial and review triage — the C28 cockpit and highlight pipeline get a salience channel that costs one envelope computation and zero causal claims.

UC 05 — Tackle-Impact Detection (the Field-Mic Event)

The practical problem: the collision fans and welfare staff care about is visible only as a pile of bodies — vision sees a ruck forming, not the impact instant. The mechanism is UC 01's transient detector one weighting down: a tackle thud is a broadband impulse with more low-frequency mass, so flux plus a low-band ratio extracts it on field-side microphones. The honest boundary belongs in the mechanism sentence: in the broadcast mix the same thud is sub-mixed beneath commentary and crowd [INFERENCE] — this is a venue-mic feature, not a broadcast one, and UC 10 exists to say so. Where a field mic or referee-adjacent feed exists, the detected impact hands C11 pose a collision frame and C30's IMU streams a cross-check timestamp.

Spectrogram of a tackle impact on a field microphone: one short broadband stripe labeled TACKLE between two colliding player silhouettes, with a sideline microphone inset.
Figure 15.12: UC 05 — The Collision Stripe. Detectable on field mics, buried in broadcast [INFERENCE]. Rugby league: tackle counts and play-the-ball window anchors for welfare and load monitoring. Pickleball: the structural analog is rare body/paddle contact faults and doubles near-collisions at the kitchen — same transient machinery, lower stakes.

Payoff: player welfare — collision counts and timestamps for load management, scoped honestly to rigs that carry a field mic.

Category III: The Fusion Layer (UC 06–08)

UC 06 — Kitchen-Firefight Audio: Rally Tempo at 250ms

The practical problem: a kitchen firefight is a burst of sub-250ms exchanges that 30fps video smears into ambiguity — the exact scenario where E04's detector wall and frame quantization compound. The mechanism needs no new detector, only a new reading of UC 01's output: inter-onset intervals. IOIn = tn+1 − tn; a firefight is a run of IOIs below a tempo threshold (~400ms [verify — practitioner threshold, pending pb-003 measurement]), and the IOI distribution inside the burst is the hand-speed curve. The same clustering segments ordinary rallies: pops chained below a gap threshold form a rally, and the first onset after the longest gap is the serve.

Spectrogram of a kitchen firefight: a dense cluster of rapid vertical stripes labeled FIREFIGHT with orange inter-onset gap brackets and a two-player volley inset.
Figure 15.13: UC 06 — The Firefight Burst. Onset density is the excitement curve; the gaps are the tempo. Pickleball: firefight detection, hand-speed proxies, and the practice-design input for C29. Rugby league: the analog is rapid tap-pass sequences and kicking duels — onset bursts on field mics [verify].

Payoff: coaching — quantifies the drill that decides kitchen battles, from a sorted list of floats instead of a tracking pipeline.

UC 07 — Audio-Video Fusion: the Multimodal Join

The practical problem: audio knows when with no semantics; vision knows what, who, where with ±16.7ms timing and an occlusion problem. Neither is believed alone for a claim that feeds the event log. The mechanism is the chapter's spine as an algorithm: onset at t proposes; vision confirms inside W = [t − 50ms, t + 50ms] using the book's own verified machinery — the E10 temporal-diff ball cue (0.814 precision, measured) or an E03 pose-velocity spike at the contact limb. The asymmetry is load-bearing: reversing the direction forces the ball detector past its demonstrated wall (median confidence 0.11, measured) before the one-dollar sensor votes. Vision is spent only where it is strong — a ±50ms crop with C09 tracking context already warm.

Two-lane fusion diagram: microphone to onset detector in the upper lane, video frames to tracks in the lower lane, meeting at a 50ms junction box feeding an event log.
Figure 15.14: UC 07 — The Join. Audio lane stamps, vision lane confirms, the ±50ms junction decides. Pickleball: contacts confirmed behind NVZ occlusion — the six pb-003 frames vision missed are the rows this closes. Rugby league: whistle-window scoping plus tackle corroboration where field mics exist.

Payoff: the C10 event log upgrades from vision-only guesses to corroborated events — the difference between a highlights guess and an officiating record.

UC 08 — Audio Event Confidence: the Two-Bit Policy

The practical problem: fusion output needs a policy for the frequent case where only one modality fires — silence is information too. The mechanism is a two-bit confidence state attached to every candidate: audio+vision is a full-confidence event feeding shot attribution (C10); audio-only stays a candidate — and is not a failure, because E04 (measured) says the ball is often invisible, so absent visual confirmation is weak negative evidence; vision-only is suspicious — a missed pop or a visual false positive — and is flagged for review; none marks the silence segments between rallies, itself useful for rest-period statistics. The construction is the book's own [verify against C10's attribution slots]; it earns its place when the §15.10 labs show the split behaving as claimed on pb-003.

Four confidence state cards in a row: A+V solid border with waveform and eye, AUDIO dashed, VISION with a flag, NONE dotted and empty.
Figure 15.15: UC 08 — The Four States and Their Policies. The audio-only card survives on E04's evidence. Pickleball: occluded kitchen contacts persist as candidates instead of vanishing. Rugby league: vision-only discontinuities inside the ruck queue for human review rather than polluting the log.

Payoff: downstream analytics honesty — C21 ratings and C22 EPV can weight every event by its confidence state instead of treating all rows as equal.

Category IV: Rigging & Honest Limits (UC 09–10)

UC 09 — Mic Placement: the One-Dollar Rig That Fixes Geometry

The practical problem: one microphone timestamps but cannot locate — and cannot reject pickleball's signature confound, the adjacent court's pop, which is the same event class in the same band. The mechanism is geometry, not spectral cleverness: 2–4 USB or phone mics at the court corners, GCC-PHAT-synchronized to sub-millisecond from ambient audio alone (Knapp & Carter 1976, pure NumPy), then TDoA-gated against the known court polygon — an arrival-time difference outside the possible range is a foreign pop. The numbers set the headroom: c ≈ 343 m/s, so 1ms of TDoA error is 34.3cm and one 48kHz sample is 7mm of sound path — sub-decimeter localization is physically comfortable [INFERENCE from geometry]. This is the Zenniz pattern (30 audio sensors + 4 cameras, verified zenniz.com) at a hundredth of the price [proposed lab extension, not yet run].

Overhead split diagram: a pickleball court with four corner microphones and TDoA arcs localizing one impact near the kitchen line, beside a rugby pitch with three sideline microphones and a wind arrow.
Figure 15.16: UC 09 — The Rig. Corner mics plus GCC-PHAT sync turn timing into location. Pickleball: the adjacent-court-bleed fix — TDoA gating to the court polygon. Rugby league: sideline arrays along the touchline for tackle-impact capture (UC 05), with windscreens against the outdoor limit (§15.8).

Payoff: the cheapest hardware upgrade in the book — a few dollars of microphones retiring a confound no classifier can solve.

UC 10 — Broadcast-Mix Separation: the Honest Limit

The practical problem: broadcast audio is not venue audio. It is a produced mix — commentary ducked over crowd, compressors flattening amplitude — and the events a fan most wants (the tackle thud) are sub-mixed beneath both [INFERENCE]. The mechanism is disciplined feature choice plus a scoped ambition: loudness features lie under compression, so whistle detection runs on spectral shape (narrowband ratio, flatness, duration) and the crowd channel runs on envelope shape; source-separation models (demucs-class) could in principle unmix stems, but sport-specific performance is unestablished [verify]. The honest engineering answer for tackle audio is not a better separator — it is a different microphone (UC 09). What broadcast audio can deliver is already large: the whistle clock (UC 03) and the crowd state (UC 04), both measured-feasible on nrl-001's lane-1 pass (107 onsets in 20s, the 4103Hz transient class).

Layered broadcast mix diagram: a thick COMMENTARY band on top, a wide fuzzy CROWD bed in the middle, and a thin faint TACKLE stripe partially hidden underneath with an orange marker.
Figure 15.17: UC 10 — The Sub-Mixed Event. The tackle exists in the mix the way a whisper exists in a crowd [INFERENCE]. Rugby league: accept whistle + crowd from broadcast; route impact capture to field mics. Pickleball: the contrast case — clean fixed-camera court audio shows the limit is the production mix, not the sport.

Payoff: scope discipline — the chapter's claims stay inside what each audio source can honestly deliver, which is what keeps the rest of the pipeline believable.

15.10d What This Adds to the Pipeline

The ten use cases are one sensor read ten ways, and each feeds an existing chapter rather than starting a project. UC 01–02 and UC 06–08 populate the C10 event log with corroborated contacts and bounces, scoped by the UC 08 confidence states that C21 ratings and C22 ΔEPV weight; UC 03's whistle windows are the compute budget for C09 re-tracking and C19 eventing in the rugby arm; UC 04's crowd channel modulates C28 cockpit review priority; UC 05 and UC 09 hand C11 pose collision frames and C30 IMU streams their sync timestamps; UC 07's ±50ms join is the C16 video-understanding alignment budget; and UC 10 keeps every one of those claims inside the honesty boundary C31's rights-and-consent framing expects. The sync discipline underneath all of it is C04's clock math, measured rather than assumed.

15.11 Sources

  • librosa (ISC) onset/spectral methods — librosa.org; PANNs CNN14 (MIT, arXiv:1912.10211); YAMNet (Apache-2.0, TF Hub); AST (MIT, arXiv:2104.01778); Phi-4-multimodal / Qwen3-Omni (audio-first models — license check before use).
  • CNN whistle detection: MDPI Applied Sciences 2022, 12(23):12151 (CC-BY, mdpi.com/2076-3417/12/23/12151). Crowd/referee evidence: Nevill, Balmer & Williams 2002 (Psychology of Sport and Exercise); Reade & Singleton closed-doors analysis.
  • Zenniz (zenniz.com, verified 2026-08-30) — 30-audio-sensor + 4-camera smart court; internals proprietary. Sync: GCC-PHAT (Knapp & Carter 1976); ITU-R BT.1359 broadcast A/V tolerance [verify exact ms].
  • Lab: lab/w3_lab_audio.pyexperiments/c15-audio/outputs/metrics.json (107 onsets/20s; 4103Hz transient class — measured).

Next Chapter

Chapter 16 — Video Understanding: Action Recognition, Spatio-Temporal

The 2026 recipe, the SoccerNet suite, and the measured gap between describing and deciding

Continue Reading
AS '26

Agentic Sport Analytics

A practitioner's field guide to automated sport analytics: watching, tagging, modelling, interpreting, and acting with AI, LLMs, computer vision, and agent harnesses. Measured on pickleball and Australian rugby league. By Mehran Mozaffari. First Edition, August 2026.

Front Matter

Preface

Front Matter

Copyright & License

Watching

Chapter 01 — Build the Lab, Not the Manuscript

Watching

Chapter 01 — Why This Book Exists: The Five Verbs of Sport Analytics

Watching

Chapter 02 — The Evidence Contract & Data Provenance

Watching

Chapter 02 — The Evidence Contract & Data Provenance

Watching

Chapter 03 — Calibrating the World: Homography & Court Geometry

Watching

Chapter 03 — Sport Rules as Formal Systems

Watching

Chapter 04 — Finding & Tracking the Actors: From ByteTrack to Meta SAM 2/3

Watching

Chapter 04 — Capture: Cameras, Lenses, Shutter, Placement

Watching

Chapter 05 — The Body in Motion: 2D Keypoints to Meta SAM 3D Body

Watching

Chapter 05 — Data Engineering for Sport Video

Watching

Chapter 06 — Smashing the Ball Wall: Spatio-Temporal Trajectory Recovery & SAM 2/3 Equipment Segmentation

Watching

Chapter 06 — Calibration I: Homography, Intrinsics, Distortion

Watching

Chapter 07 — The Structured Representation: PBN & State Machines

Watching

Chapter 07 — Calibration II: Broadcast Dynamics, GMC, and Per-Frame H_t

Tagging

Chapter 08 — Reading Space & Pressure: Geometric Deep Learning

Tagging

Chapter 08 — Detection: YOLO, RF-DETR, and the AGPL Decision

Tagging

Chapter 09 — Generative Replay & Counterfactual Simulation

Tagging

Chapter 09 — Tracking & Identity: Metrics, ReID, and Role Priors

Tagging

Chapter 10 — Where Vision-Language Models Help, and Where They Lie

Tagging

Chapter 10 — Segmentation & Foundation Models: SAM 2/3, DINOv3

Tagging

Chapter 11 — Building the Live Coaching Cockpit on Apple Silicon

Tagging

Chapter 11 — The Body in Motion: 2D Keypoints to 3D Biomechanics

Tagging

Chapter 12 — Complex Motion & Field Sport Scaling

Tagging

Chapter 12 — Smashing the Ball Wall: Spatio-Temporal Trajectory Recovery

Tagging

Chapter 13 — Evaluation, Rights, and the Next 10 Runs

Tagging

Chapter 13 — Identity: Who Is Who

Tagging

Chapter 14 — Multi-Camera Geometry, Line Calls, 3D Reconstruction

Tagging

Chapter 15 — Audio & Multimodal Cues: The Free Sensor

Tagging

Chapter 16 — Video Understanding: Action Recognition, Spatio-Temporal

Tagging

Chapter 17 — Event Data & the Common Representation

Modelling

Chapter 18 — Annotation: The Ground-Truth Workflow

Modelling

Chapter 19 — Automatic Eventing: State Machines, Confidence, Review Queues

Modelling

Chapter 20 — Statistics for Sport Practitioners

Modelling

Chapter 21 — Rating Systems: DUPR, ELO, Glicko, and Skill

Modelling

Chapter 22 — Expected Value: xG, VAEP, EPV, and Their Sport Transplants

Modelling

Chapter 23 — Tactical ML: Graphs, Equivariance, and Honest Forecasting

Modelling

Chapter 24 — Simulation & Counterfactuals: The Honest Rebuild

Interpreting

Chapter 25 — Where Vision-Language Models Help, and Where They Lie

Interpreting

Chapter 26 — From Numbers to Narrative: Reports, Scouting, Coach UX

Interpreting

Chapter 27 — The Agent Harness for Sport Analytics

Interpreting

Chapter 28 — The Live Coaching Cockpit: Real-Time Systems, Honestly Measured

Acting

Chapter 29 — Practice Design & Interventions: The Acting Loop

Acting

Chapter 30 — Sensors & Hardware: Wearables, Smart Courts, the Fusion Spine

Acting

Chapter 31 — Deployment, Licensing, Rights & Ethics

Acting

Chapter 32 — The Laboratory: Reproducing the Book's Claims

Acting

Chapter 33 — The Frontier: What's Changing in 2025-2026

Acting

Chapter 34 — The Book as a System: How to Use It (Human + Agent)

©2026 Mehran Mozaffari. Free for personal/noncommercial use (CC BY-NC-ND 4.0); commercial license required for business use.