life-recorder: Owning the Ambient Capture Pipeline With an iPhone and a Mac

The x-axis of lifelogging: why this is a capture problem, not a transcription problem

Everyone evaluates lifelogging systems on the wrong axis. They compare transcription quality, retrieval accuracy, summarization pipelines — the y-axis of the problem. The real axis, the one that determines whether a system gets used at all, is capture surface: what physical hardware is listening, and what happens to the audio between the microphone and the transcript.

life-recorder by browser-use makes a specific, opinionated bet on that axis, and understanding it requires seeing the four paradigms it sits between.

The commercial wearable camp — Limitless, Plaud, Bee, the Humane pin — solves capture by removing the phone from the equation entirely. A dedicated pendant means no iOS background-execution fights, no battery sweat on your phone, far-field microphones designed for pockets and rooms. The cost is that every second of audio goes to a vendor cloud for transcription and vector storage. You pay twice: once for hardware, once monthly, and again in the currency of ambient audio you'll never fully audit leaving your possession.

The open-source hardware camp — Omi, OpenPendant — keeps the wearable form factor but lets you point audio at whatever endpoint you configure. You trade a soldering-adjacent setup and a BLE bandwidth ceiling for sovereignty, but you still bought and now have to charge a second device.

The desktop recorders — Screenpipe, the Rewind/Limitless Mac lineage — are the opposite failure. They never leave your machine, which is excellent, and they are structurally blind to your commute, your walking meetings, your kitchen-table conversations. Screen context with no mobility.

life-recorder picks a fourth position: your existing iPhone, your existing Mac, nothing else. The innovation is not in the Whisper inference — whisper-cli running GGML models is thoroughly solved. It's not in the transcript format. The bet is that the marginal friction of ownership (zero new hardware, zero subscription) and the hard guarantee of locality (audio crosses your LAN over TLS and is deleted post-transcription) matter more than any transcription-pipeline improvement. It's a different point along the capture-surface axis, and it's the one I find most defensible for personal use.

Dimension	life-recorder	Commercial wearables	Open-source wearables	Desktop passive recorders
Capture surface	Native iPhone background mic	Dedicated pendant / magnetic device	Dedicated BLE hardware module	Host machine mic + system audio
Compute location	Local Mac (whisper.cpp, GGML)	Vendor cloud (Whisper/AssemblyAI/etc.)	Configurable: cloud endpoint or self-hosted server	Local on the same host
Mobility coverage	High — wherever the phone goes	High — wherever the pendant is worn	High — within wearable range	Low — desk-bound only
Data output	Flat chronological Markdown (life.md)	Structured meetings, action items, search UI	Graph memory, raw transcripts, daily notes	SQLite with vector embeddings, own API

The table makes the trade explicit: you're accepting a flat file and a build step in exchange for owning every layer and paying nothing recurring. Whether that's the right trade depends entirely on whether you can tolerate the failure modes that come with it — which is where the mechanics actually get interesting.

The hard mechanics: chunked AAC, a durable delivery queue, and a single append-only file

The pipeline is worth walking precisely, because the elegance and the brittleness live in the same design choices.

On the iPhone, the app holds an AVAudioSession open continuously and captures into roughly one-minute AAC files. The chunking is not an optimization — it's the retry unit. A one-minute AAC file at typical bitrates is a small, self-contained object; if it fails to upload, the cost of re-sending is trivial. Each chunk lands in a local queue on the phone that persists until the Mac receiver has acknowledged durable receipt. This is the key mechanical decision: the phone does not assume delivery, and it does not delete on transmit. It deletes on acknowledgment. Upload runs over authenticated HTTPS with a random bearer token stored in the iOS Keychain, paired against a self-signed TLS certificate. Stale network tasks get cancelled so a dead request can't block the queue behind it.

On the Mac, a Python receiver accepts the chunk, issues the durable acknowledgment, and hands the audio to whisper-cli running a local GGML model — ggml-small.bin is the reference point. Transcription output goes through a regex post-processing pass that strips stage-direction artifacts and the repetitive hallucination patterns Whisper is known for on silence and ambient hum. The cleaned text is appended to life.md, a single Markdown file organized with hourly capture markers. Then, immediately, the audio chunk is deleted.

flowchart TD
    A[iOS AVAudioSession<br/>continuous background capture] --> B[~1-minute AAC chunk files]
    B --> C[Local queue on iPhone<br/>persists until ack]
    C --> D[Authenticated HTTPS upload<br/>bearer token + self-signed TLS]
    D --> E[Mac receiver<br/>Python + whisper.cpp]
    E -->|durable ack| C
    E --> F[whisper-cli transcription<br/>local GGML model]
    F --> G[Regex text filtering<br/>strip stage directions + hallucination artifacts]
    G --> H[Append to life.md<br/>with hourly timestamps]
    H --> I[Immediate audio chunk deletion]

Why this is elegant: the durability semantics are correct. Acknowledgment-before-deletion means the queue is the source of truth, and a crashed receiver just means chunks pile up on the phone rather than vanish. The output being a plain append-only file means any agent that can read a file can consume it — no proprietary API, no database driver, no schema migration. And immediate deletion means the privacy claim is structural, not a policy promise: raw voice exists for seconds.

Why it is brittle: every one of those clean properties has a sharp edge. The fixed one-minute boundary slices mid-sentence, so chunk seams produce garbled fragments and the regex filter can't repair meaning. Deletion-after-transcription means there is zero ground truth audio left to check a hallucinated sentence against — you've optimized so hard for privacy that you've discarded your only verification path. And the queue that makes delivery robust also makes storage unbounded: disconnected for a multi-day trip, that's hundreds of megabytes to gigabytes of AAC accumulating on the phone, with no backpressure beyond iOS eventually killing you for it. The design is correct about the hard things and quietly exposed on the soft ones.

The way the pieces actually fit: receiver as a listening daemon, iPhone as a dumb uplink

The architectural inversion here is the part I'd copy in other systems. The Mac receiver is the supervising node. It owns the transcription model, it owns the durable acknowledgment protocol, it owns the external output artifact. The iPhone is a capture-and-upload client with exactly two jobs: keep the microphone alive and don't lose a chunk until told to. That split is not accidental — it maps to what each device can actually do well.

The phone can only solve local problems. It can hold a queue, retry an upload, cancel a stale task. It cannot run useful Whisper inference without cooking its battery, and it cannot assemble context across days of audio. The Mac is where compute and context assembly belong: it has the silicon for fast inference, it has the disk for a growing transcript, and it's the machine your agents already run on. Pushing the intelligence to the supervising node and keeping the endpoint dumb is the correct direction of dependency.

receiver/setup.py is the deployment surface for that supervisor. It generates the self-signed TLS certificate, creates the pairing page and bearer token, and can install a background launch agent so the receiver survives reboots and runs as a daemon rather than a terminal you forgot to reopen. The launch agent is the difference between a toy and an always-on service — without it, every Mac restart silently ends your lifelogging. SKILL.md in the repository is the other half of the setup story: a standard skill file that agentic coding tools can read to drive the Xcode build, the receiver configuration, and the launch-agent installation themselves. Given that the setup friction here is genuinely high — Xcode, an Apple Developer team, a signed build — handing that work to an agent that can execute a documented procedure is the right mitigation.

sequenceDiagram
    participant Dev as Developer / Agent
    participant Xcode as Xcode + Apple Dev Team
    participant Phone as iPhone (LifeRecorder)
    participant Setup as receiver/setup.py
    participant Mac as Mac Receiver Daemon
    participant Agent as Downstream Agent Tool

    Dev->>Xcode: Provision & sign app
    Dev->>Setup: Run setup.py
    Setup->>Setup: Generate self-signed TLS cert + bearer token
    Dev->>Phone: Install & launch app
    Phone->>Mac: Pair handshake (token from Keychain) + TLS trust
    Mac-->>Phone: Pairing acknowledged
    Setup->>Mac: Install background launch agent
    Note over Phone,Mac: Pair/ack completes before any audio is transmitted
    Mac->>Mac: Receiver daemon listening
    Phone->>Phone: Start continuous AVFoundation capture
    Phone->>Mac: Upload ~1-min AAC chunks (HTTPS + bearer token)
    Mac-->>Phone: Durable receipt acknowledged
    Mac->>Mac: whisper-cli transcription + regex filtering
    Mac->>Mac: Append to life.md with hourly timestamps
    Agent->>Mac: Read life.md as context

The handshake before audio matters: no chunk leaves the phone until the token pairing and TLS trust are established, which means an unconfigured receiver can't quietly accumulate audio it wasn't meant to hold. The daemon then becomes a passive listener — chunks arrive, transcripts append, agents read. The phone never needs to know what's downstream.

Where this actually breaks: silent death, storage pressure, and the sleep problem

The first failure mode is the one that quietly corrupts your dataset rather than crashing anything: iOS will not restart continuous background microphone capture on its own after a reboot or a force-quit. If the phone auto-updates overnight, runs flat, or you swipe the app away, recording simply stops, and it does not come back until you physically unlock the device and open the app. There is no headless restart path — the OS forbids it by design. What that means operationally is that your 24/7 lifelog has silent gaps, and the gaps are invisible in the artifact: life.md just has less text for those hours, with hourly markers that skip cleanly past the dead window. If downstream agents treat the transcript as a complete record, they'll reason over a timeline with holes they can't see. The practical mitigation is a watchdog outside the phone — something that notices the stream has gone quiet and pings you to relaunch — because the phone itself cannot tell you it stopped capturing. The tradeoff is you've reintroduced a human-in-the-loop step, which is exactly the property that made "always-on" appealing in the first place.

The second is storage pressure on the uplink. When the Mac is unreachable — you're away from the LAN, the receiver is down, or you're on a multi-day trip without a tunnel — chunks accumulate on the phone with no ceiling. At typical AAC bitrates that's roughly 700 MB to 1.4 GB per day, entirely dependent on connectivity, and the queue that makes delivery durable is the same queue that now has no backpressure. The remedy is a mesh VPN like Tailscale or WireGuard so the phone has a route home over cellular, but that trades battery for reachability and adds a network dependency you now have to keep alive alongside everything else.

The third is the Mac itself. macOS sleeps when idle or when a laptop lid closes, and in sleep the receiver drops connections and the transcription daemon stops. A backlog that accumulated during sleep then has to drain on wake, and on anything less than Apple Silicon that drain can fall behind real time. The fix is caffeinate, or a genuinely always-on headless host, or an Apple Silicon box kept awake with sleep disabled. All three convert "free" into "a machine you're now responsible for keeping up."

Whisper, the 1-minute boundary, and the hallucination tax

The fixed one-minute slicing is a delivery decision that becomes a transcription liability. Cutting mid-sentence or mid-word means each chunk is transcribed in isolation, so sentence fragments appear at every seam with broken punctuation and no cross-chunk context to repair them. The text still reads plausibly enough that you notice the damage only when you're trying to extract something precise from a specific moment — which is the worst time to discover it.

The deeper problem is hallucination. Small GGML models, and ggml-small.bin is the reference configuration here, are notorious for emitting confident repetitive prose during silence, the steady hum of a fan or an AC unit, or the muffled friction of a phone in a pocket. The failure isn't random noise you can spot; it's well-formed sentences, thank-you-for-watching boilerplate, cycle-through-subtitle artifacts in random languages — things that look exactly like transcript. The regex post-processing pass catches known stage-direction markers and the patterns people have already catalogued. It cannot catch a plausible-sounding hallucinated sentence, because on the surface there's nothing wrong with it.

Then there's the compounding choice: the audio is deleted immediately after transcription. That's the privacy guarantee, and I respect it, but it means there is no ground truth left to audit against. When a hallucinated paragraph lands in life.md, you have no way to go back and listen, because the audio that would tell you no longer exists. You've optimized the verification path out of the system.

A voice-activity-detection gate before slicing, plus overlapping windows instead of hard one-minute cuts, would materially fix both the seam problem and a good chunk of the silence hallucination — VAD simply doesn't hand silence to Whisper, and overlap gives the model the run-up it needs to finish a sentence. Why it isn't trivial: VAD needs a second model in the pipeline, overlap means you're transcribing some audio twice and need de-duplication at the seams, and both additions push compute onto the Mac that now has to keep up with real time. It's the right fix, and it's a real engineering project, not a regex tweak.

The monoculture problem: why life.md is not an agent memory

Here's the thing I keep coming back to: life.md is a capture artifact, not a memory system, and the gap between those two is where most people's lifelogging ambitions quietly die. The file does exactly one job well — it appends a chronological, deterministic, plaintext record with hourly markers. Everything an agent actually needs from memory, it does not do.

The arithmetic is unforgiving. Continuous 24/7 transcription at conversational density produces somewhere in the neighborhood of thousands of lines per day. Nobody's feeding that wholesale into a context window at useful fidelity — you'll blow the budget on a single day, and the moment you truncate to fit, you've silently dropped the part where the useful thing was said. There's also no speaker diarization anywhere in the pipeline, and that's not a small omission. The receiver does monolithic mono transcription: you, the person across the table, the podcast playing in the kitchen, and the TV in the next room all land in life.md as one undifferentiated stream of text. An agent reading that will attribute a news anchor's sentence or a colleague's opinion to you, and it has no way to know it's wrong. Ambient audio doesn't announce itself as ambient audio.

So the honest framing is that life-recorder is an ingestion layer, and the integration milestone people think is the finish line is actually the starting line. The real system you have to build sits downstream: partition life.md into daily or hourly files so a single read isn't catastrophic, split those into sentence-level chunks, embed them with a local model, and land them in a time-aware vector index where recency and timestamp are first-class filters rather than metadata you bolt on later. Only after that does "my agent has memory of my day" become a true statement rather than a hopeful one.

I'd treat the single append-only file as a deliberate design tradeoff rather than a flaw. It's trivially inspectable, greppable, and agent-readable, which is why it's the right v0 output. But it's a source of truth for humans and pipelines, not a retrieval substrate, and the sooner you stop asking it to be your agent's memory, the sooner you build the layer that can be.

Putting it to work: three project builds around a local capture backbone

The right way to think about life-recorder is as a backbone you hang real systems off. It gets audio off your person and into text on a machine you control. Everything interesting happens after that, and each build below picks a different downstream problem worth solving.

Event-sourced daily memory. The first build keeps your agent's context small by giving it a summary instead of a log. A Python service watches life.md for new hourly blocks, splits the day's blocks into sentence-level units, embeds them with a local model, and — critically — queries only the current day's context when an agent session starts. It appends a structured daily summary to a second file with explicit date-time references, so the agent loads the summary and reaches for the raw transcript only when it needs verbatim detail. The wiring is life-recorder's receiver output feeding a file watcher, a sentence splitter, a local embedding model, and a small SQLite database keyed by timestamp and embedding, exposed to the agent as a tool that reads today's summary. The gotcha is the upstream boundary: that fixed one-minute chunking leaves sentence fragments at every seam, so you want VAD-based merging or a sliding-window overlap on transcription before you split, and you want the daily summary annotated with confidence flags so the agent doesn't treat a hallucinated stretch of silence as a remembered fact.

Geofenced privacy shutter. The second build is the one I'd prioritize before anything clever. On the iOS client, add a companion toggle that checks location against a user-defined list of private zones — home, conference rooms, a clinic — and stops the AVAudioSession on entry, so life.md records a short mute marker instead of audio. Concretely: iOS CoreLocation region monitoring writes zone state to shared preferences or an HTTP endpoint on the receiver, and the capture client consults it before each chunk upload, emitting a muted marker when suppressed. The failure mode to design against is that iOS location permissions and background updates are separately throttled, and region monitoring is not guaranteed while the app is in a low-power state — so make failure conservative. Default to stop capture unless an explicit allow rule matches. A shutter that fails open is worse than no shutter.

Calendar-aware meeting context scrambler. The third build bridges the receiver to your calendar. Match meeting start and end times from a CalDAV endpoint against the hourly timestamps in life.md, mark those windows as meeting contexts, and let a downstream semantic search return only verbatim snippets from your own spoken turns with a speaker label attached. The components: EventKit or CalDAV on the Mac side, a parser aligning meeting windows to transcript blocks, a light classifier or heuristics to separate your turns from ambient speech, and a local FAISS index over just those snippets. Be honest about the limitation — there's no diarization upstream, so your speaker heuristic only approximates identity. Don't lean on attribution for anything legal or sensitive. And when the phone is out of LAN range, time correlation degrades because chunks arrive late and out of order, which means your meeting windows can drift.

All three share the same shape: life-recorder owns capture and locality, you own meaning. That division is the whole point of the architecture, and it's why I'd rather build on this than rent a closed pipeline.

Resources

• GitHub - browser-use/life-recorder · GitHub
• raw.githubusercontent.com