Living dossier

3D & Gaussian Splatting

Back to topics•Blog
Mehran Mozaffari·
9 resources0 related posts

What 3D Gaussian Splatting Actually Is — After the Hype Cycle Settled

Gaussian splatting is a scene representation: millions of small, coloured, anisotropic 3D ellipsoids, each with an opacity, that together reconstruct what a camera saw. Trained from posed photos or video, the resulting model renders photorealistic novel views at real-time rates — 30 frames per second or better at 1080p in the original 2023 result — because rendering it is closer to point-cloud rasterisation than to neural-network inference. That mechanical property, not the picture quality alone, is why splatting displaced NeRFs in production conversations: a NeRF needs a neural network evaluated per ray, while a splat scene is data that ordinary graphics hardware can draw.

Three facts frame everything else in this dossier. First, the reference implementation is INRIA's gaussian-splatting repository — Kerbl, Kopanas, Leimkühler, and Drettakis, ACM Transactions on Graphics 2023 — and its training requirements still define the floor: a CUDA GPU with compute capability 7.0+, and 24 GB of VRAM to train to paper-evaluation quality. Second, the official code ships under a bespoke Inria/Max Planck research licence that permits research use and evaluation — commercial deployment of the original code is not what the licence grants — while the ecosystem's re-implementations mostly sidestep this with Apache-2.0 or MIT licences. Third, the format war is effectively over at the capture end and still open at the delivery end: .ply from the trainer, then compressed derivatives (.splat, .ksplat, .spz, .sog) depending on which viewer you target.

What keeps getting miscategorised into this field: single-image-to-3D generation (a different problem — no multi-view capture, far weaker geometric guarantees), photogrammetry mesh pipelines (the deliverable is a mesh, not splats), and text-to-CAD tools (parametric engineering geometry, a completely different output contract). They belong to the same "3D from AI" conversation and often share capture hardware, but their runtime, editing, and licensing realities differ enough that treating them as one market produces wrong tool choices.

The Production Pipeline, End to End

A splat that actually ships passes through six stages, and the failure modes live at every boundary:

  1. Capture — photos or video from a phone, drone, or camera rig. Quality here dominates everything downstream; the capture path, overlap, and exposure consistency matter more than the training algorithm.
  2. Posing — COLMAP (structure-from-motion) recovers camera intrinsics, poses, and a sparse point cloud that seeds the Gaussians. The official trainer consumes COLMAP output directly.
  3. Training — 30 minutes to hours on a CUDA GPU; the optimiser interleaves densification (splitting and cloning Gaussians) with rendering loss, guided by spherical harmonics for view-dependent appearance.
  4. Cleanup and compression — floaters get culled, then the model is converted and compressed for its delivery target: .splat for web viewers, .ksplat for the Three.js renderer, .spz for Niantic's format, compressed .ply for Spark.
  5. Serving — a static file behind CORS, a viewer library in a web page, or a native runtime.
  6. Interaction — the consumer orbits, walks, or embeds the scene alongside regular meshes.
flowchart TD
    CAP["Capture: photos or video"] --> COLMAP["COLMAP posing<br/>camera poses + sparse points"]
    COLMAP --> TRAIN["Training: 3DGS optimiser<br/>24 GB VRAM class for paper quality"]
    TRAIN --> PLY["Trained model: .ply with millions of splats"]
    PLY --> CLEAN["Cleanup: cull floaters, crop bounds"]
    CLEAN --> COMPRESS["Convert and compress:<br/>.splat / .ksplat / .spz / compressed .ply"]
    COMPRESS --> SERVE["Serve: static file behind CORS or CDN"]
    SERVE --> VIEW["View: WebGL or WebGPU viewer"]
    VIEW --> EDIT["Optional: edit, animate, embed alongside meshes"]

The training runtime itself has consolidated. The gsplat library — the nerfstudio project's open-source CUDA rasteriser — now reproduces the official implementation's PSNR, SSIM, and LPIPS exactly while using up to 4x less GPU memory and up to 15 percent less training time, and its 2026 changelog shows where the research energy went: sparse rasterisation with active-tile rendering, multi-GPU dense training, LiDAR and fisheye camera models, 3DGUT trajectory support, MCMC densification strategies, and an inference-only fast path (HiGS) that packs scenes into fp16 for low-latency web rendering. Nerfstudio wraps gsplat into a config-driven trainer (its splatfacto method) with a web viewer, which is the friendliest on-ramp for teams without CUDA engineering.

sequenceDiagram
    participant Dev as Developer
    participant T as Trainer (nerfstudio / INRIA)
    participant V as Converter
    participant CDN as Static host (CDN)
    participant B as Browser viewer

    Dev->>T: COLMAP dataset
    T->>T: Optimise Gaussians (hours, GPU)
    T-->>Dev: Trained .ply
    Dev->>V: Convert to .splat / .spz / .ksplat
    V-->>Dev: Compressed artifact
    Dev->>CDN: Upload behind CORS headers
    B->>CDN: Fetch splat file (progressive)
    B->>B: Decode, sort splats, render per frame
    B-->>Dev: Navigable scene in the browser
    Note over B,CDN: Sorting cost and memory live client-side -<br/>this is where delivery fails on mobile

The Trainer Layer: Reference Code Versus Productised Stacks

The INRIA reference implementation remains the ground truth: it defines the metrics everyone reproduces, ships pre-trained models and the SIBR viewers, and gains occasional research features — training-speed acceleration, depth regularisation, anti-aliasing, exposure compensation, OpenXR viewing. Position: where you go for fidelity to the paper and for research baselines. Its costs are the 24 GB VRAM target, a CUDA-only CUDA-extension build, and the research-only licence.

gsplat (Apache-2.0) is the production rasteriser most modern trainers sit on. It is not a tool you point at photos — it is the CUDA layer — and its role in the stack is the reason the ecosystem could diversify licences and platforms at all. Position: the default engineering choice; if you are building anything splat-shaped yourself, start here.

Nerfstudio (Apache-2.0) is the friendly trainer wrapper: modular NeRF and splat methods, web viewer, Colab support, and community maintenance. Position: the fastest legitimate path from capture to trained model for teams without graphics engineers — you trade some control for the config system and viewer.

Brush is the outlier that quietly matters most for accessibility: a 3D reconstruction engine in Rust on WebGPU (via the Burn framework) that trains splats natively on macOS, Windows, Linux, Android, and in a browser — no CUDA dependency, no Python, dependency-free binaries. It takes COLMAP or Nerfstudio-format data, supports masking, streams .ply over a URL, and visualises training live. Position: the strongest answer to "does splatting have to mean CUDA", with browser training still gated to Chrome and Edge for WebGPU support.

GaussianGPT (ECCV 2026 oral) represents the generation frontier: an autoregressive transformer that generates 3D Gaussians by next-token prediction — a VQ-VAE compresses per-voxel Gaussians into discrete tokens, a causal transformer with 3D rotary embeddings models them, and decoding re-renders through gsplat. Unlike diffusion-based 3D generation, the autoregressive formulation supports completion, outpainting, temperature-controlled sampling, and flexible horizons. Position: the clearest research evidence that splats are becoming a token format for generative models, not just a capture artefact — with generation quality still behind capture-based scenes for photorealism, as the authors themselves frame it.

The Delivery Layer: Where Splats Actually Reach Users

antimatter15/splat is the artifact that proved browser delivery was possible: a WebGL 1.0 implementation with no dependencies, unminified and readable, with progressive loading, drag-and-drop .ply conversion, and CPU splat sorting in a web worker. Its honesty is its documentation: it drops spherical harmonics to shrink files (third-degree SH would be nearly 200 bytes per splat), and it acknowledges the sorting problem is unsolved-in-principle. Position: the reference for understanding how splat delivery works, and still the lightest viewer to deploy.

GaussianSplats3D brought splats into the Three.js ecosystem — .ply, .splat, and a custom compressed .ksplat, WebXR support, octree culling, WASM and GPU-assisted sorting — and its README now says plainly that it is no longer in active development, recommending Spark instead. Its documented limits are the best public spec of the web delivery envelope: roughly 16 million splats at SH degree 0, falling to about 8 million at degree 2; CPU sorting artifacts when moving fast; large scenes crashing the splat sort. Position: respect its documentation as the field's delivery-constraint reference; choose its successor for new work.

Spark (World Labs, MIT) is that successor: a Three.js renderer built for fusing splats with mesh content, targeting 98-percent-plus WebGL2 device support, rendering multiple splat objects with correct mutual sorting, supporting the major formats (.ply including compressed, .spz, .splat, .ksplat, .sog), and — the capability that marks where this is going — fully dynamic splats: per-splat transforms, colour editing, displacement, skeletal animation, and a shader-graph system for GPU-side editing. Position: the current default for shipping splats in web products, backed by a company (World Labs) whose entire thesis is spatial intelligence, which makes its maintenance risk unusually low.

The comparison that matters when choosing delivery:

Viewer / renderer Licence Runtime Formats Distinctive strength Main limitation
antimatter15/splat Unlicensed code, MIT-styled minimal WebGL 1.0, zero dependencies .ply (converts), .splat Lightest possible deployment; readable source No spherical harmonics; CPU sort
GaussianSplats3D MIT-style (check repo) Three.js, WebGL .ply, .splat, .ksplat WebXR, octree culling, WASM sort Inactive; CPU sort artifacts
Spark MIT Three.js, WebGL2 (98%+ devices) .ply, .spz, .splat, .ksplat, .sog Dynamic editable splats, multi-object sorting, mobile-first Newest; Three.js-coupled
Brush viewer Rust engine (GPL-family; verify per use) WebGPU (Chrome/Edge), native, Android .ply, .compressed.ply, animation zips In-browser training, not just viewing WebGPU browser gate

Cost Shape and the Licence Trap

The compute cost curve is the opposite of most AI fields: inference is nearly free (a browser renders splats), and training is the bill. INRIA-reference quality wants a 24 GB CUDA GPU for hours; gsplat cuts memory up to 4x and time up to 15 percent; Brush moves training onto consumer hardware and even browsers at some quality cost. There is no per-token cost — the recurring costs are capture labour, GPU rental for training (minutes to hours per scene), storage and CDN bandwidth for multi-hundred-megabyte uncompressed scenes or single-digit-megabyte compressed ones, and engineering time for cleanup. Compression choice directly prices delivery: a full SH-degree-3 .ply runs hundreds of megabytes; a trimmed .splat or .spz lands in the tens; .ksplat tuning can cut further at fidelity cost.

The licence trap is the field's most repeated mistake. The original INRIA/MPI implementation is research-only — non-commercial use is what the licence grants — so the code most tutorials teach is code you may not ship commercially. The production-legal paths are the re-implementations: gsplat and nerfstudio (Apache-2.0), Spark (MIT), Brush and antimatter15/splat (check the exact file for your use). Trained splat assets themselves occupy a grey zone that production teams resolve by training with a permissively licensed implementation, which is now also the faster one.

Capture Discipline: Where Quality Is Actually Won or Lost

The unglamorous truth of this field is that capture quality predicts final quality better than any training choice, and the reference pipeline's dependence on COLMAP makes capture a hard prerequisite rather than a quality knob. The practical rules that separate good scenes from ghost-infested ones: dense angular coverage with 60 to 80 percent overlap between neighbouring views; consistent exposure (auto-exposure during capture is the most common invisible defect — the optimiser bakes exposure inconsistency into the Gaussians, which is why the official trainer and gsplat both grew exposure-compensation features); sharp frames only, because motion blur becomes smeared Gaussians; and no moving objects, since anything that moves between frames trains double geometry.

Scale changes the rules again. Mip-NeRF 360-style outdoor scenes, interior rooms, drone captures of buildings, and object-isolated turntables each have distinct failure envelopes, and the reference evaluation splits them deliberately (MipNeRF360, Tanks and Temples, Deep Blending) for exactly this reason. A team's first splat project should pick the capture envelope closest to one of those benchmark families before improvising.

Editing, Animation, and the Mesh Boundary

The oldest criticism of splats — you cannot edit a cloud of ellipsoids — is being answered on three fronts at once. Spark ships per-splat transforms, real-time colour editing, displacement, skeletal animation, and a GPU shader-graph system, which makes splats behave more like ordinary scene content inside Three.js. GaussianGPT's autoregressive formulation makes completion and outpainting native operations: because scenes are token streams, the model can continue a scene or fill a masked region the way language models continue text. And Brush's masking support — transparency and mask folders that constrain which pixels influence training — gives capture-time control over what becomes geometry.

The boundary with meshes is also softening from both sides. Spark renders splats and meshes together with correct sorting, so a product can put a photoreal captured environment around traditionally modelled interactive objects. GaussianGPT's VQ-VAE treats Gaussians as a compressible token space, which is the representation-level version of the same convergence. The remaining hard case is precision: splats carry no topology, so anything needing exact edges, booleans, or manufacturing tolerance — the CADAM and text-to-CAD corner of the 3D field — still ends in meshes or parametric geometry, not ellipsoids.

Choosing by Constraint in the Splat Stack

The constraint-driven selection: if fidelity to the paper dominates, INRIA's reference implementation on a 24 GB GPU. If engineering pragmatics dominate, gsplat plus nerfstudio — Apache-2.0, reproducible metrics, less memory, less time. If no CUDA anywhere dominates, Brush, accepting the WebGPU browser gate and some quality cost. If web delivery to ordinary devices dominates, Spark with a compressed format, testing against the splat-count ceilings that GaussianSplats3D's documentation quantifies. If generative or completion workflows dominate, GaussianGPT's checkpoints are the research front, used as research rather than production. And if commercial shipping dominates every other answer, the licence column, not the feature column, makes the first cut: research-only code is disqualified before the demos load.

Where It Breaks: Failure Modes and Their Triggers

COLMAP failure on hard surfaces. Trigger: textureless walls, reflective or transparent objects, repetitive patterns, moving objects during capture. Posing fails or produces sparse seeds in the wrong places, and training amplifies the error into double-walls and ghost geometry. There is no post-hoc fix that is cheaper than recapturing with better overlap and texture.

Under-capture at boundaries. Trigger: too few views of object sides, tops, and interior corners. The model fills the gaps with stretched, blurry Gaussians that look acceptable from training views and collapse from novel angles. Mitigation is capture discipline — overlap, orbits, deliberate side coverage — not algorithmic.

Floaters and sky artefacts. Trigger: unconstrained captures where the sky, moving foliage, or semi-transparent surfaces let the optimiser place semi-transparent ellipsoids in empty space. Cleanup is a mandatory pipeline stage (the cleanup step exists in every production stack), and tools that skip it ship visible garbage.

Memory and splat-count ceilings in the browser. Trigger: uncontrolled scenes with tens of millions of splats delivered to a web viewer. GaussianSplats3D's documented envelope — about 16 million splats at SH degree 0, fewer with view-dependence — and its note that very large scenes crash with index-out-of-bounds errors are the concrete limits; mobile devices fall well below them. The answers are compression, SH-degree reduction, cropping, and LOD — all of which cost fidelity.

Sort-induced swimming. Trigger: fast camera motion in viewers whose splat sort runs on the CPU or at limited precision. Splat rendering requires per-frame depth sorting; imprecision or latency shows as popping and swimming, which the GaussianSplats3D documentation ties directly to integerBasedSort and distance-map precision settings. This is why GPU-side sorting and incremental sorting are standing feature requests.

The training-rendering divergence. Trigger: teams evaluating quality on training views. A scene can look perfect from captured viewpoints and fall apart on novel ones. The only honest evaluation is held-out views scored with PSNR/SSIM/LPIPS — which the reference implementation ships (full_eval.py, about seven hours on an A6000 for the standard suite) and gsplat reproduces exactly.

The licence tripwire. Trigger: shipping a product whose trainer was the INRIA reference code. The research-only licence does not grant commercial deployment, and the discovery usually arrives at the worst time — after capture and training spend. Choosing gsplat/nerfstudio/Spark from day one eliminates the class of problem.

Open Questions the Field Has Not Settled

Whether generative splats reach capture quality is the frontier question GaussianGPT's oral at ECCV 2026 formalises — autoregressive splat generation with completion and outpainting is architecturally compelling, but no public result yet claims photorealistic parity with multi-view capture on arbitrary scenes. Whether dynamic-scene splatting (people moving, objects deforming) gets a standard pipeline — 3DGUT, 4D zips in Brush, per-splat animation in Spark are early answers — or fragments per use case is undecided. Whether the delivery format question resolves at all is genuinely open: .spz, .sog, .ksplat, and compressed .ply each have ecosystems, and Spark's decision to support all of them is both pragmatic and an admission that the market has not chosen. Whether training leaves the data centre entirely depends on WebGPU compute shipping in more browsers than Chromium, which Brush's browser gate makes visible in real time. And the field still lacks a shared quality measure for the things operators care about — geometric stability under motion, editability, delivery weight — since PSNR and LPIPS measure view fidelity and nothing else. Each of these will be settled by shipped products rather than papers, which is why this dossier tracks repositories as closely as it tracks research.

Resources

Foundations and reference implementation

Training and rasteriser stacks

Generative frontier

Delivery and viewers