What I Learned Stitching Together Homography, Tracking, and Temporal Detection in a Custom Vision Pipeline

The Pipeline I Kept Rebuilding: Why Modular CV Still Wins

The naive version of this pipeline takes a weekend. Detection to pose to tracking to homography to event detection is five well-documented stages, each with mature open-source implementations, and wiring them together in a Python script is genuinely a few hundred lines. The version that survives contact with real broadcast footage takes months, and almost none of that time goes into any single stage. It goes into the seams.

The core problem is that this architecture is a chain of stages where each one is designed under the assumption that its input is clean. The detector emits a bounding box and assumes the frame is representative. The pose estimator crops inside that box and assumes the box is tight and correctly centered. The tracker consumes keypoints and assumes they're temporally coherent. The homography projection consumes foot positions and assumes they're noise-free and on the ground plane. The event detector consumes metric coordinates and assumes the trajectory is smooth. Every one of those assumptions is false in production, and the failure propagates forward and amplifies.

What makes it worse is that upstream errors are often invisible downstream. A dropped detection for two frames doesn't look like a dropped detection; it looks like a sudden velocity spike. A jittery ankle keypoint doesn't look like jitter; it looks like a player teleporting half a meter. A slightly wrong homography doesn't look like a calibration problem; it looks like the whole pitch is subtly wrong in a way that only shows up when you compare distances against ground truth.

Here's how the cascade actually flows:

graph LR
    D[Detector] --> P[Pose Extractor]
    P --> T[Tracker]
    T --> H[Homography Projection]
    H --> E[Event Detector]

    D -.->|detector miss: box absent or wrong| P
    D -.->|detector miss: stale association| T
    P -.->|pose jitter: bad ground contact point| H
    T -.->|ID switch: trajectory attached to wrong person| H
    H -.->|homography instability: metric coords explode| E
    T -.->|ID switch: velocity derivative corrupted| E
    E -.->|event boundary ambiguity: false peak| E

I've found that the practical response is not to pick better models in each slot. It's to build the pipeline as if every stage is hostile, which means validation layers between every pair of stages, and that's where the real engineering lives.

That framing is what this piece is about: the specific ways each stage lies to the next one, and what I'd actually do about it.

Temporal Event Detection: The Boundary Between Fuzzy and False

This is the stage where everything upstream gets judged, which is unfortunate, because it's also the stage with the least tolerance for upstream noise. Event detection is nominally a classification problem — did the ball bounce, did the foot cross the line, did contact occur — but in practice it's a localization problem. The label is easy. The boundary is hard.

The older approach — frame-by-frame heuristics or sliding-window 3D-CNNs (I3D, SlowFast) — treated the timeline as a sequence of independent windows and asked "is this an event?" per window. That produces exactly the failure you'd expect: no consistent notion of where an event starts and ends, so the same bounce gets labeled across four overlapping windows with four different confidences. Action transformers like ActionFormer and TriDet changed the framing by predicting explicit start/peak/end boundaries over the full temporal extent, which is why I'd reach for them over 3D-CNNs when the boundary position actually matters downstream. A bounce that's localized to ±200ms is usable. A bounce that's "somewhere in this two-second window" is not.

The boundary jitter problem is real and mostly physical. At 30 or 60 fps with rolling shutter, a fast-moving ball or a swinging limb smears across rows of the sensor within a single frame, and the frame where the bounce "happens" is genuinely ambiguous — not because the model is weak, but because the sensor integration window overlaps the event. A ball-bounce versus a footstep at the same pixel location can produce nearly identical motion signatures across two frames. This is where I'd stop expecting a single monocular classifier to nail it and start using temporal context: a bounce has a distinct post-event trajectory (elastic rebound), a footstep doesn't. Feeding the trajectory shape, not just the local window, into the boundary decision is the difference between a detector that fires on every motion blip and one that fires on the event.

Here's the sequence as it actually runs, including where the false positive gets born:

sequenceDiagram
    participant V as Video (30/60 fps)
    participant D as Detector
    participant T as Tracker
    participant H as Homography
    participant W as Window Generator
    participant C as Event Classifier

    V->>D: frame t0
    D->>T: detections at t0
    V->>D: frame t1
    D->>T: detections at t1
    Note over D,T: detector drops t1 for the ball
    V->>D: frame t2
    D->>T: detections at t2
    T->>H: associated tracks (gap at t1)
    Note over T: Kalman covariance spikes on re-association
    H->>W: metric ground-plane coords (jittered near t1)
    W->>C: candidate window around motion signature
    Note over W,C: jitter creates a spurious peak
    C->>C: classify window boundaries
    C-->>W: FALSE POSITIVE (phantom bounce)

The key insight is that the false peak was manufactured two stages earlier. The classifier did its job correctly on a corrupted input.

The Failure Cascade, Quantified: What Breaks First and How to Detect It

Treat the cascade as a directed graph where each node's output is the next node's input distribution, and the failure modes are the ones that shift that distribution without raising an alarm at the node where the shift originates.

A detector miss at a single frame does three distinct things. First, the pose extractor receives a stale box — most top-down estimators (MMPose, ViTPose, RTMPose) crop within the box and assume it's current, so a pose extracted against a one-frame-old box can land keypoints on the wrong limb or off the body entirely. Second, the tracker's association step now has a gap, and the Kalman posterior covariance on position and velocity inflates before re-association. On a fast-moving object, a single-frame gap can produce a velocity estimate that's off by a factor of two or more. Third, and worst, that inflated velocity flows into the homography projection as a near-discontinuity in ground-plane coordinates, which the event detector sees as a legitimate trajectory anomaly — the phantom bounce in the sequence above.

The mitigations I'd actually put in place, in priority order:

Never feed raw homography output to downstream logic. This is the single highest-leverage guard. Insert a validation layer that rejects any projected coordinate that (a) exceeds a plausible max speed for the object class — say 15 m/s for a thrown ball, 12 m/s for a sprinting human — (b) falls outside the known field/court boundary by more than a margin, or (c) has a first difference that exceeds N standard deviations of the recent trajectory. Rejected points get replaced by a short-horizon prediction, not by the bad value and not by a gap.

Smooth with a moving average tied to the actual frame timestamps. Not a fixed-window smoother — the real dt from the RTSP stream, because variable frame rates and dropped frames are the norm, and a smoother that assumes uniform spacing will miscalculate derivatives even on clean data.

State-machine fallbacks per stage. When the detector's confidence distribution shifts (more low-confidence boxes than usual), flag DETECTOR_MISS and hold the last good track rather than emitting a degraded one. When the tracker reports an ID switch, force a resync before letting the trajectory into the homography. When homography drift is detected — typically via a line-registration check against a known field marker — trigger an online recompute from whatever reference features are visible.

Synthetic calibration data for the edge cases you'll never collect. Empty venues, extreme low-angle lighting, heavy shadows across the field, partial occlusions of the corner markers. Feature-based homography dies specifically in these regimes, and you cannot test for them with a clean daytime clip.

The state diagram of how I'd actually wire the guards:

stateDiagram-v2
    [*] --> NOMINAL

    NOMINAL --> DETECTOR_MISS: confidence histogram shifts left
    NOMINAL --> SYNTHETIC_CALIBRATION_REQ: feature count below threshold
    NOMINAL --> TRACKER_RESYNC: ID switch reported

    SYNTHETIC_CALIBRATION_REQ --> NOMINAL: regeneration passes validation
    TRACKER_RESYNC --> NOMINAL: association confirmed over K frames

    DETECTOR_MISS --> POSE_JITTER: stale box fed to pose extractor
    POSE_JITTER --> TRACK_ID_SWITCH: wrong keypoints corrupt association
    TRACK_ID_SWITCH --> HOMOGRAPHY_DRIFT: bad ground contact points projected
    HOMOGRAPHY_DRIFT --> EVENT_FALSE_POSITIVE: phantom trajectory anomaly detected

    HOMOGRAPHY_DRIFT --> ONLINE_RECOMPUTE: line-registration check fails
    ONLINE_RECOMPUTE --> NOMINAL: recomputed H passes boundary sanity

    EVENT_FALSE_POSITIVE --> NOMINAL: bounds-check validation rejects event

Each recovery arrow points back to a healthy state, and the discipline is that no stage is allowed to promote itself back to NOMINAL on its own say-so — the guard above it has to agree.

Stitching vs. Single-Stage: Where the Tradeoff Budget Really Lies

The honest framing of this comparison is that it's not really "modular vs. single-stage." It's "which parts of the problem do you need metric precision on, and which parts can tolerate semantic fuzziness." Those are different budgets, and conflating them is how teams pick the wrong architecture.

The modular custom pipeline — YOLO-family or RT-DETR detection, RTMPose/ViTPose pose, ByteTrack/BoT-SORT/OC-SORT tracking, DLT+RANSAC or a deep field-registration model for homography, an action transformer for events — wins on one axis decisively: it runs at 60–120+ FPS on consumer or edge hardware (Jetson, a single consumer GPU) when you compile to TensorRT or ONNX Runtime. That's the throughput budget that live broadcast and instant-replay decisions actually demand. It also wins on the precision budget when the geometry is known and the camera is calibrated, because the DLT/RANSAC solve is mathematically exact for planar points and doesn't hallucinate. Its cost is integration overhead and the cascade I described above.

Specialized frameworks sit in the same tier but skip the boilerplate. OpenMMLab's ecosystem (mmpose + mmtracking + mmaction2) and Roboflow Sports / Supervision give you pre-packaged field registration, tracker motion compensation, and annotation codecs. If I were starting today and the domain were sports, I'd seriously consider building on these rather than reinventing tracker GMC from scratch — the tradeoff is fine-tuning against your specific camera views and accepting their dataset schema.

Dense point/trajectory models are a genuinely different bet. SAM 2 does persistent object segmentation across video; CoTracker and PIPs track arbitrary points through occlusions and non-linear motion without hand-engineered motion priors; 4D-Humans recovers SMPL-X bodies directly in metric space, sidestepping the separate detection → 2D pose → 3D lifting chain. These are the right tools when motion is deformable and non-linear and you can't afford to hand-tune a Kalman filter for every object class. The cost is compute: most of these run sub-realtime on a single consumer GPU, and critically, they do not give you world coordinates without external planar anchors. A dense tracker tells you where a point went in image space; it doesn't tell you it moved 2.3 meters across the court.

Video-VLMs are the tier where the precision budget and the semantic budget get confused most often. Gemini 2.x/1.5 Pro, Claude 3.5/3.7 Sonnet, and GPT-4o can ingest native video and do zero-shot temporal event detection and high-level reasoning without any stitching. That's real and useful. They cannot do metric homography, cannot track a coordinate at 60fps consistently, and will confidently hallucinate an exact boundary line or timestamp. Their token cost is also prohibitive for continuous stream monitoring — you burn the budget watching empty frames at the same rate as the frame that matters.

Turnkey multi-camera hardware (Hawk-Eye, Second Spectrum, KinaTrax/Qualisys) sits at the top of the precision budget and the bottom of the accessibility budget: 6–12+ synchronized high-speed cameras, multi-view triangulation, calibrated extrinsics giving sub-millimeter accuracy. It's the officiating standard. It's also multi-million-dollar fixed infrastructure that's useless for single-camera footage.

The hybrid approach that's actually emerging in production is the one I'd bet on, and it splits the budget cleanly. Run detection, pose, tracking, and homography locally in an optimized runtime — TensorRT or ONNX Runtime — and feed the resulting structured telemetry, sequences of (X, Y, Z, t) per tracked object, to a VLM asynchronously for semantic verification and summarization. That's the shape of a VLM-assisted sports metric verifier: the VLM never sees raw frames, never does metric work, and answers questions like "did the ball bounce before the line?" by reasoning over structured coordinates. When the VLM disagrees with the metric pipeline, you flag for human review — you don't let the model override the DLT solve. And you batch events, not frames, or you'll spend the entire inference budget on the 99% of frames where nothing happens.

The same split shows up in a cost-aware VLM fallback for temporal events: run a heuristic or lightweight action transformer locally to generate candidate event windows, run an ambiguity classifier on those windows, and only call the VLM for the genuinely ambiguous cases. Keep local latency for obvious events, pay VLM tokens only where they buy accuracy. The failure mode to watch for is exactly the cascade this whole piece is about — if upstream tracker jitter is producing phantom motion peaks, your local detector overfires, your ambiguity classifier stays busy, and your VLM bill scales with your tracker's instability rather than with actual event volume. The confidence threshold on the local classifier is load-bearing; set it too low and you've rebuilt the cost problem you were trying to solve.

There's also a quieter case for an investment that pays off across the whole stack: a synthetic edge-case calibration harness. A rendering tool that produces empty venues, extreme lighting, heavy shadows, and partial occlusions gives you ground-truth-labeled data to test each stage independently — homography validation against known field geometry, detector robustness against adverse visuals, tracker ID-switch stress under occlusion. The catch is honesty: don't overfit to synthetic visuals. Blend the synthetic suite with a small, curated set of real edge-case clips so the regression tests stay tethered to the distribution you'll actually deploy against. I've watched synthetic-only validation pass cleanly while the production pipeline fell apart on a single sun-shadow across the baseline.

The throughline across all of this is that the precision budget and the semantic budget are different currencies. Spend local compute on the geometry, spend model tokens on the meaning, and build guards at every seam between them. The basketball ReID stack I wrote about argues the same thing from the tracking side, and the RF-DETR/BoT-SORT/VLM walkthrough is the concrete version of the hybrid split for a domain where the field geometry is fixed and the semantic questions are rich.

The Production Fallback Menu: What I'd Actually Ship First

When I'm integrating this pipeline into something with a real deadline and a real budget, the question stops being "what's the best architecture" and becomes "what can I drop without breaking the business logic." The answer is almost always the same: the geometric spine is load-bearing and the semantic tail is evictable.

Detection and ground-point homography stay. Nothing downstream exists without a box, and the DLT solve against known field geometry is the thing that turns pixels into meters. Pose-to-ground-contact is the fragile part of the spine — if I'm time-constrained, I'll take the ankle/toe keypoints and reject the bounding-box-center shortcut that causes the parallax blowup, even if that means a slightly weaker pose model. What I can push out the door fastest is precise temporal event detection. A rolling-window heuristic on metric trajectory (velocity sign flip plus rebound direction = bounce) is worse than an action transformer on the boundary, but it's honest about being fuzzy and it doesn't require a trained classifier on day one. The VLM verification layer can come later, on the ambiguity cases the heuristic flags, not on every frame.

graph TD
    A[Raw video in] --> B[Detector - LOAD BEARING]
    B --> C[Pose / ground contact point - LOAD BEARING]
    C --> D[Tracker - LOAD BEARING]
    D --> E[Homography DLT solve - LOAD BEARING]
    E --> F[Guard layer - LOAD BEARING]
    F --> G[Temporal event detection - EVICTABLE at v1]
    G --> H[VLM semantic verify - DEFERRED, async]

The guard layer is not optional even in the minimum viable version. Max-speed bounds checking, boundary rejection, and timestamp-aware smoothing cost almost nothing to write and prevent the entire cascade from reaching whatever business logic is consuming coordinates. If homography coordinates flow directly into a scoreboard or a betting feed, one tracker ID switch becomes a wrong number with real consequences, and the guard is the only thing standing between you and that.

Stage	Load-bearing?	Under-pressure compromise	Acceptable degraded mode
Detection	Yes	YOLO-family instead of RT-DETR; skip fine-tuning on your cameras	Slightly lower recall, more dropped frames absorbed by tracker coasting
Pose / ground point	Yes, but fragile	Ankle/toe keypoints only; skip full body topology	Missing limbs OK, ground contact must be present
Tracker	Yes	ByteTrack instead of BoT-SORT; skip GMC if camera is static	ID switches increase; downstream guard must catch them
Homography	Yes	DLT+RANSAC with manual corner marks instead of auto field registration	Recalibrate on camera bumps; drift caught by line-registration check
Guard layer	Yes	Even a few hundred lines of bounds and speed checks	None — this is the last line of defense before business logic
Temporal events	No (v1)	Rolling-window heuristic on metric trajectory	Boundary error ±300ms; false positives on ambiguous motion
VLM verification	No (v1)	Defer entirely; add async once heuristic ambiguity rate is measured	No semantic backstop; still ship the metric pipeline

The throughline: ship the metric spine with guards, ship the heuristic event layer with an honest boundary error budget, and let the VLM come later as an async verifier on flagged cases. The RF-DETR walkthrough is the concrete version of this staged rollout for a fixed-geometry domain.

Where I'd Push Next: When to Stop Stitching and Start Querying

The direction with the most genuine leverage is collapsing the detection → 2D pose → 3D lifting chain into a single-stage metric body regressor. 4D-Humans and SMPL-X regressors infer bodies directly in metric space without the intermediate boxes and keypoints, which removes two of the seams where my cascade lives. If I don't have a detector that can drop a frame, I don't have a stale box feeding a pose crop. The cost is compute — these don't run at 60fps on the edge today — but the trajectory is clear and the seam-count reduction is real.

Dense point persistence is the other axis I'd watch. CoTracker and PIPs track arbitrary points through occlusion and non-linear motion without a Kalman filter tuned per object class, which is exactly the hand-engineering I resent. Where I'm skeptical is the same place I've been skeptical throughout this piece: none of these give you world coordinates. A dense tracker that survives a full occlusion still only tells you where a point went in image space. You still need a planar anchor and a homography solve to say it moved 2.3 meters.

The field-registration models are quietly the highest-ROI advance for the domains where geometry is known. Deep homography estimators that predict calibration directly from image features — handling pan and zoom automatically — kill the feature-based drift that dies under occlusion, shadow, and extreme zoom. If I were rebuilding my sports pipeline today, that's the swap I'd make first.

So the principle I'd hold onto as the frontier moves: use a VLM when you need to know whether something happened semantically, in context, under rules that are easier to state in language than in code. Use a geometric pipeline when you need to know where it happened in millimeters, at frame rate, with an auditable derivation. The two questions don't collapse into each other, and the architectures that pretend they do are the ones that hallucinate boundary lines at 60fps and charge you for the privilege.

Resources

• 301 Moved Permanently
• Welcome to MMPose’s documentation! — MMPose 1.3.2 documentation