Living dossier

Computer Vision

Back to topics•Blog
Mehran Mozaffari·
10 resources0 related posts

Computer Vision Became a Composition Problem, Not a Model Problem

The defining change in this field is not any single model. It is that the hard question moved from "which model detects best" to "which combination of models, glue, and deployment targets survives real cameras." A production perception stack in 2026 is a composition: a capture source with its own failure physics, a detector or segmentation model chosen per class of decision, a tracking or state layer that fuses frames, increasingly a vision-language model for the open-ended cases the specialist models cannot name, and a glue library that moves data between all of them with consistent formats. Every one of those stages can pass a benchmark and still sink a deployment, which is why the glue layer — once an afterthought — is now where hiring happens.

The second structural change is that the specialist-versus-generalist question became live. Foundation-style vision models (SAM for promptable segmentation, Florence-2 for multi-task prompting, Gemini-class VLMs for open-ended scene reasoning) now cover tasks that used to require a custom-trained detector. But specialist models still win on latency, cost per frame, and measurable accuracy for fixed class sets. Deciding which side of that line a workload belongs on is the central engineering judgment in applied vision today, and it is a judgement about economics as much as accuracy: a YOLO-class detector on an edge device costs fractions of a cent per thousand frames, while a frontier VLM costs orders of magnitude more for the privilege of not knowing its own failure modes.

Detection: One Family Owns the Edge, Another Owns the Frontier

Real-time detection is the most commoditised layer, and the commoditisation has a shape. The Ultralytics YOLO family — now at YOLO26 — spans detection, instance and semantic segmentation, classification, pose estimation, and built-in tracking across video, with a one-line install (pip install ultralytics), a CLI (yolo predict model=yolo26n.pt source=...), and export paths to essentially every edge target. Its licence is the commercial fork in the road: AGPL-3.0 for open use, with an Enterprise Licence offered for teams whose distribution terms conflict with AGPL. That licence is not a footnote — it is the reason alternatives keep getting funded.

The transformer challenge to that franchise is Roboflow's RF-DETR, built on a DINOv2 backbone with neural-architecture-search-found sizes. Its published benchmark table puts RF-DETR-N at 48.4 COCO AP (50:95) at 2.3 ms on a T4 with TensorRT FP16, scaling to RF-DETR-L at 56.5 AP at 6.8 ms — the README's own table shows every N-through-L size exceeding the YOLO11 and YOLO26 equivalents on both COCO metrics at comparable latency, with the licence split doing interesting work: the N/S/M/L sizes and the segmentation checkpoints are Apache-2.0, while the XL and 2XL sizes fall under a separate PML licence. And the field keeps producing research-grade challengers: DART converts Meta's SAM3 into a real-time multi-class open-vocabulary detector by sharing one class-agnostic ViT-H backbone across all text prompts, reaching 55.8 AP on COCO at 15.8 FPS (four-class, 1008 px) on a single RTX 4080 — a reminder that the open-vocabulary frontier is converging on real-time from the research side.

Position summary, stated plainly: Ultralytics YOLO26 is the safe operational default with the biggest deployment ecosystem and the AGPL string attached; RF-DETR is the accuracy-per-millisecond challenger whose Apache-2.0 core makes it the better licence fit; DART-class open-vocabulary systems are what you reach for when the class list keeps changing and a specialist fine-tune cannot keep up.

Promptable Segmentation Changed the Economics of Labels

Meta's Segment Anything Model and its SAM 2 successor changed what segmentation costs. SAM produces high-quality masks from point or box prompts, zero-shot, trained on 11 million images and 1.1 billion masks; SAM 2 extends the same promptable interface to video with streaming memory for real-time processing, and Meta's data engine turned the model into a labeling flywheel — model-assisted annotation, human correction, better model. The practical consequence for production teams: masking, background removal, region-of-interest extraction, and dataset labeling are no longer custom training projects. The auto-mask generator gives whole-image segmentation in a few lines; the ONNX export path puts the promptable decoder in browser and edge runtimes.

The economics deserve the emphasis. Before SAM, instance segmentation on a new domain meant thousands of hand-drawn masks or a fine-tune cycle. After SAM, the workflow became model-proposes-human-corrects, and annotation volume dropped by multiples on typical scenes. The remaining costs are compute (the ViT-H encoder is the heavy part of the two-stage design, which is why edge deployments distill or swap encoders) and the evaluation work of confirming that zero-shot masks meet your quality bar rather than merely looking right in demos.

flowchart TD
    CAM["Capture: camera, video file, or stream"] --> DETECT["Detection model<br/>YOLO26 / RF-DETR / DART-style"]
    CAM --> SEG["Promptable segmentation<br/>SAM / SAM 2"]
    DETECT --> TRACK["Tracking and state fusion<br/>ByteTrack-style association"]
    SEG --> ANNOT["Auto-labeling and region extraction"]
    ANNOT --> TRAINLOOP["Fine-tune detection on corrected labels"]
    TRACK --> REASON["Task logic: zones, counts, events"]
    REASON --> GLUE["Glue layer: consistent detections format"]
    GLUE --> DOWN["Downstream: alerts, dashboards, robotics, OCR handoff"]
    DETECT --> GLUE

Depth and Foundation Models Turned Single Images Into Sensors

Monocular depth estimation crossed from research curiosity to utility with the Depth Anything line. Depth Anything V2 ships four scales — 24.8M-parameter Small through a 1.3B Giant — for robust relative depth, with metric-depth variants distilled from the Small and Base checkpoints, integration into Hugging Face Transformers and Apple Core ML, and successor projects (Video Depth Anything for temporally consistent long videos, Prompt Depth Anything for 4K metric depth when low-resolution LiDAR is available to prompt the model). The engineering significance: relative depth from a single RGB frame is now a cheap, reliable sensor input for robotics, AR occlusion, photo effects, and safety checks — no stereo rig required.

On the understanding side, Microsoft's Florence-2 is the small-model argument for vision-language tasks. It is a prompt-based foundation model — caption, object detection, segmentation, OCR — in 0.23B and 0.77B sizes under an MIT licence, trained on 5.4 billion annotations across 126 million images (the FLD-5B dataset), with a continued-pretraining variant extending context to 4k. A model of that size running on modest hardware covering captioning through OCR through detection is the reason "just use a VLM" is now a legitimate architectural answer for open-ended perception — and its COCO OD AP of 39.8 for the large variant tells you honestly where it stands against specialist detectors: remarkably close for one model, still behind the specialists at the frontier.

The frontier-lab side kept moving too: the SAM 3 line extends zero-shot segmentation to 3D and video without per-video fine-tuning, Gemini-class flash models compete on the cost of image analysis at scale, and Qwen3-VL offers an open-weight vision-language alternative for teams that need self-hosted multimodal reasoning. The composition consequence: open-ended understanding is becoming a cheap component, which pushes specialist models toward the high-rate, fixed-class, latency-critical slots where they remain unbeatable.

OCR Quietly Became Document Intelligence

The OCR corner of vision had its own step change, and it came from PaddleOCR. The project — 70k-plus GitHub stars, Apache-2.0, used by thousands of downstream repositories — now pitches itself as document-intelligence infrastructure: PP-OCRv6 recognises text in 50 languages with one unified model, reports +4.6 percent detection and +5.1 percent recognition over its predecessor while surpassing mainstream vision-language models with a 34.5M-parameter medium tier, and claims a 5.2x CPU inference speedup end to end. Above the OCR layer, PP-StructureV3 converts complex PDFs and images into Markdown or JSON with fine-grained coordinates, and the PaddleOCR-VL-1.6 vision-language model (0.9B parameters) posts 96.3 percent on OmniDocBench v1.6 with particular strength on formulas, tables, seals, and historical documents. The HPD-Parsing companion pushes throughput to thousands of tokens per second via hierarchical parallel decoding.

Position: for LLM-era pipelines that need documents as structured context, PaddleOCR is the default open-source answer, and its Apache-2.0 licence plus CPU-friendly deployment make the total cost roughly the cost of the CPU it runs on. The competitor set is now VLM-based document parsing, which wins on messy layout generalisation and loses on throughput, determinism, and price per page — a trade that should be measured per document class, not assumed.

The Glue Layer Decides Whether Any of It Ships

The least glamorous layer is the one that most often determines success. Roboflow's supervision library is the field's recognition that detection outputs need a common currency: it is model-agnostic (connectors for Ultralytics, Transformers, MMDetection, and Roboflow Inference), and it provides the reusable tools — annotation, zone counting, line crossing, tracking wrappers — that every vision pipeline needs but nobody wants to rewrite. Apache-2.0, Python 3.9+, one pip install.

OpenCV remains the substrate: the Apache-2.0-licensed 4.x library that every capture path, codec, and classical vision primitive bottoms out in, with the opencv_contrib module set carrying the long tail. MediaPipe is Google's on-device answer for the mobile and web side: pre-built solutions for object detection, pose, hand and face landmarks, and segmentation that deploy to Android, iOS, web, and edge with a Model Maker path for customisation — the fastest route when the deployment target is a phone and the task matches a stock solution.

The glue layer's real contribution is contract standardisation. When detections from any model arrive in one schema, thresholds, zone logic, tracking, and alert policies become portable across model swaps — which converts "we retrained the detector" from a breaking change into a non-event. That portability is also what makes the evaluation work (below) possible at all.

A Comparison of the Named Options on Real Axes

Tool / model Task focus Licence Runs where Cost shape Position
Ultralytics YOLO26 Detect, segment, pose, track AGPL-3.0, Enterprise paid Edge to cloud, one pip install Free under AGPL; Enterprise Licence for closed distribution Operational default; licence is the fork
RF-DETR Detection, segmentation Apache-2.0 (N-L); PML 1.0 (XL/2XL) CUDA/TensorRT targets Free core; T4-class latency published Accuracy-per-ms challenger
DART Open-vocabulary real-time detection SAM research licence inherited Single consumer GPU, TensorRT FP16 Free code, non-commercial terms Research frontier for changing class lists
SAM / SAM 2 Promptable segmentation, video Apache-2.0 code; check checkpoint terms GPU, ONNX edge Free; labeling-cost collapse is the value Labeling economics redefined
Depth Anything V2 Monocular depth Open weights on Hugging Face CUDA, MPS, CPU; Core ML Free; relative depth default, metric needs prompting Depth as cheap sensor input
Florence-2 Caption, detect, segment, OCR MIT 0.23B/0.77B params, modest hardware Free weights; cheapest VLM-class option Small-model multi-task generalist
PaddleOCR / PP-OCRv6 OCR and document parsing Apache-2.0 CPU-first, GPU, NPUs Free; CPU-cheap at production volumes Document intelligence default
supervision Glue: annotation, zones, tracking Apache-2.0 Anywhere Python runs Free; the integration work it replaces is the cost The connective tissue
MediaPipe On-device detection, pose, landmarks Apache-2.0 Android, iOS, web, edge, Python Free; device-bound Mobile and web deployment default
OpenCV Capture, classical CV, codecs Apache-2.0 (4.x) Everywhere Free; assumed infrastructure The substrate everything stands on

Open Clients, Free Tiers, and Cheap Small Models Reshaped the Toolkit Inside a Year

Three recent changes altered real decisions. RF-DETR's YOLO-beating benchmark table with an Apache-2.0 core gave teams with AGPL aversion a first-class real-time alternative, and its transformers integration moved fine-tuning into the standard Hugging Face workflow. PP-OCRv6's unified 50-language model with CPU speedups made multilingual OCR a single-model deployment rather than a model zoo, which changes the cost of multilingual document pipelines materially. And the VLM price-performance slide — Florence-2-class small models covering caption-detect-segment-OCR on modest hardware, with flash-tier commercial VLMs competing on cost for scale image analysis — moved open-ended scene questions from "fine-tune a specialist" toward "ask a generalist," at least for the low-rate slots in a pipeline. The SAM 3 announcement of zero-shot 3D and video segmentation extends the promptable-segmentation franchise to the temporal dimension; teams building video annotation flows should evaluate it against SAM 2 before building custom trackers.

The Auto-Labeling Loop That Replaced Annotation Budgets

The workflow that SAM-style promptable segmentation made standard deserves a precise description, because it is the loop most vision teams now run instead of hiring annotators:

sequenceDiagram
    participant O as Operator
    participant M as Detection model (current)
    participant S as SAM / SAM 2
    participant H as Human reviewer
    participant T as Training set

    O->>M: Run on new capture batch
    M-->>O: Candidate detections with confidence
    O->>S: Prompt segmentation per candidate region
    S-->>O: Pixel-accurate masks
    H->>S: Correct mask edges where wrong
    H->>T: Approve corrected labels
    T-->>M: Fine-tune or retrain on expanded set
    Note over M,T: Each cycle raises the floor -<br/>the model proposes better next round
    M-->>O: Deploy improved model to edge targets

The loop's economics: SAM-class models collapsed mask-annotation cost by multiples, auto-labeling tools built on them collapsed bounding-box cost similarly, and the residual human cost concentrates exactly where it should — on the frames the model finds hard, which are the frames worth a human's time. The two discipline points that keep the loop honest: sample the low-confidence detections for human review rather than discarding them, and hold out a fixed evaluation set from the training loop so "improvement" is measured on data the model has never influenced.

Where It Breaks: Failure Modes and Their Triggers

Lighting and domain drift. Trigger: models evaluated on clean benchmark imagery deployed against real cameras with backlighting, night scenes, weather, or sensor noise. Accuracy degrades silently — no crash, just worse detections that downstream logic trusts. The mitigations are capture-representative validation sets, per-camera thresholds, and drift monitoring on input statistics, none of which any model card provides for your specific cameras.

The generalist hallucination tax. Trigger: vision-language models answering open-ended questions about scenes. VLMs describe objects that are not present, miscount, and infer rather than observe — failures that a specialist detector structurally cannot make. Use VLMs for open-ended slots with verification; use specialists where the class set is fixed and the decision is consequential.

Small-object and dense-scene collapse. Trigger: high-resolution frames with small targets — aerial imagery, retail shelves, industrial defects — fed to models trained on COCO-scale objects. Detection recall collapses. Mitigations are tiling with overlap, higher input resolutions at latency cost, and model sizes chosen for the object scale, not the leaderboard.

The AGPL compliance surprise. Trigger: Ultralytics YOLO embedded in a distributed product without reading the licence. AGPL obligations reach network services, and the commercial licence is the intended path. This surfaces at acquisition due diligence or after launch, both expensive moments. RF-DETR's Apache core exists precisely for this buyer.

Edge-target export drift. Trigger: models exported to TensorRT, Core ML, or ONNX runtimes with different preprocessing than training. Accuracy drops with no error message. The fix is unglamorous — golden-image regression tests that compare exported-model outputs against the training framework on a fixed image set, run in CI on every export.

Tracking identity switches. Trigger: crowded scenes where ByteTrack-style association hands IDs between similar objects crossing paths. Downstream counting and event logic then double-counts or drops objects while the per-frame detection metrics look perfect — the failure lives between frames, invisible to single-image evaluation. Mitigations: motion-consistency gates on ID assignment and evaluation metrics that measure track continuity, not just frame accuracy.

Multispectral and thermal blind spots. Trigger: pipelines designed for RGB deployed on thermal or multispectral sensors, where pretrained weights have never seen the modality's statistics. Fine-tuning helps; assuming transfer does not. The correct default is to treat a new modality as a new project with its own capture-representative evaluation, even when the model architecture carries over.

OCR confidence mirage. Trigger: OCR output that looks clean but contains character-level errors (confusable digits, homoglyphs) inside otherwise-correct text. Downstream LLM pipelines ingest the errors as facts. Mitigations: confidence-thresholded routing to human review, cross-field validation (checksums, known vocabularies), and never letting a single OCR pass be the only source for consequential numbers.

Open Questions the Field Has Not Settled

Whether open-vocabulary detection can beat fine-tuned specialists at edge latencies is the question DART and its successors are racing to answer — the capability argument is won, the latency argument is not yet. Whether VLMs replace specialist pipelines for high-volume workloads is an economics question being settled by invoices, not papers: the crossover depends on per-frame VLM prices continuing their slide, and the answer will differ per workload class. How video understanding standardises — SAM 3's zero-shot video segmentation, Video Depth Anything's consistency, and per-frame detector stacks are three incompatible architectural answers — remains open. And the evaluation gap is structural: benchmarks measure single-frame accuracy, while deployments fail on temporal consistency, drift, and edge-case economics, and no accepted public benchmark measures what operations actually need. Until one does, the field's working rule is the oldest one in applied machine learning: hold out your own data, measure your own failure costs, and let your deployment constraints — not a leaderboard — pick the stack.

Resources

Detection and segmentation models

Depth and foundation models

OCR and document intelligence

Glue, deployment, and substrate