llama.cpp · Parakeet · local MCP bot-less recorder · 1Password
the cloud never commands it
The desktop client
Tauri 2 — React 19 webviews + Rust core. Not Electron. The edge runtime lives in Rust precisely because the WebView freezes its JavaScript in the background (v0.19.0 lesson) — inference and tools survive a minimized window.
Signed installers: Developer ID + notarization (macOS), Authenticode EV (Windows), .deb/.rpm (Linux). minisign updates, publicly verifiable manifest.
Browser-based device pairing (no token pasting); token in the OS keychain, out of devtools' reach; session refreshed and the request replayed on 401 instead of kicking you to the login screen.
The recorder
Detection by the mic's hardware state, not the calendar or a process scan: a CoreAudio listener (macOS — catches Meet in a browser tab), PulseAudio introspection (Linux), the ConsentStore registry (Windows).
Debounced state machine with a 2-second "Arming" state — because recording the lobby is exactly what you don't want. An open-but-idle Zoom client triggers nothing; the status glyphs Chrome grafts onto tab titles (🔊 🔴) are stripped before classification.
Per-process auto-stop (macOS 14.4+): "is a process OTHER than me capturing?" — hanging up stops the recording even if the Meet tab stays open.
Dual-stream mic + system audio merged into stereo WAV (one channel each); mute writes silence to preserve time alignment; anti-feedback by excluding its own audio (excludesCurrentProcessAudio).
Durable upload queue (atomic manifest, retry/backoff, per-session dedup) + an append-only journal that is never purged — a chain of custody for recordings, which matters when keeping client communications is a regulatory obligation.
The edge node — AI on your machine
llama.cpp as a per-platform sidecar (Metal / CUDA / Vulkan), a catalog of 9 GGUF models pinned by SHA-256 (2.7 → 22 GB) — Qwen3.5/3.6 (4B → 35B), Ornith 1.0, among others. Streamed download with rolling hash and atomic rename. What runs on what: see the local models guide.
Local transcription with Parakeet ONNX via sherpa-onnx (+ Canary, Cohere), opt-in diarization, per-chunk checkpointed jobs (an interruption re-transcribes nothing), 100% Rust audio decoding (symphonia — zero ffmpeg).
Honest capability gating: a node only advertises inference_capable if the model is on disk AND the server resolvable — "why isn't my local agent answering?" is made impossible by construction.
The cloud never commands the machine: the backend can neither choose which model the node loads nor which MCP binaries it runs — command lines are composed locally, by design.
Headless CLI ("edge light") for servers: SSH-pairable, hot-reloaded TOML config (comments preserved), SHA-256 self-update, installable as an OS service.
Memory and self-improvement — the full system
No fine-tuning anywhere: everything runs on PostgreSQL. It's the deepest part of the product, so here it is in full.
Writing
After each AI reply, a light LLM pass extracts at most 5 durable memories per turn (durability test: "still true in 30 days?"). An empty list is the normal, expected outcome.
3-layer deduplication before every insert: exact SHA-256 hash → FTS rank ≥ 0.5 (paraphrases) → vector cosine ≥ 0.85 (paraphrases with no shared words). A duplicate re-confirms the original instead of piling up.
Anti-hijack by design: the extractor refuses to store imperatives ("a stored directive is replayed in every future context") and speaker identity; it falls back to private on any doubt. The model proposes, the code applies: no LLM writes to the database directly — validated JSON, categories clamped to the canonical enum, pinned by tests.
Every memory carries its contributor (human ≠ AI, never conflated) and its scope: company / team (Keto sharing) / personal / ephemeral — and ephemeral really is: nothing derived is persisted.
Reading
Hybrid storage tsvector + pgvector, recall by RRF fusion (k=60, 50 candidates per side), weighted by freshness decay (7-day half-life, 0.7 floor).
Honest freshness: last_confirmed_at (the fact was re-stated) ≠ last_used_at (a mere recall) — two distinct timestamps drive the decay.
An always-injected core memory tier under a hard character budget (1,200 company / 800 personal): an addition that overflows is rejected. Scarcity is the curation mechanism — predictable preamble, no context drift.
Accent-folded FTS on both sides ("prefere" typed without the accent matches the stored "préfère") and extraction in the conversation's language — bilingualism is in the schema, not a patch.
At night
Dream sweep at 4:30 UTC: deterministic SQL pruning first (old + never recalled + rarely used, durable categories exempt), then batched LLM re-judging (keep/archive with typed reasons), merging redundancies, resolving contradictions (superseded_by), and re-scoping personal leaks back to private — distinguishing "a fact about a third party" (stays with the team) from "a personal fact about the contributor" (goes back private). The curator reorganizes; it never invents and never DELETEs.
Resurrection: an archived fact that gets re-stated in conversation is automatically un-archived — it cleared the selective extractor again, so it's evidently alive.
The loop over skills and prompts
No judge-and-party (dated product decision, 2026-06-05): the agent that produced a reply never judges it. A third-party observer only re-reads conversations flagged by deterministic triggers (👎, abnormal length) and records friction facts only.
Observer → proposer → author → apply → revert pipeline: pre-mutation journal, bit-for-bit revert, the revert reason becomes feedback, the reverted triple is blacklisted. The dashboard measures whether the fix got used (activations_since) and whether the friction stopped (recurrence_since).
Tools and secrets
43 MCP modules on a single FastMCP server: 21 external integrations (Slack, Teams, Gmail, Jira, GitHub, HubSpot…) + 22 platform modules (documents, memory, todos, boards, telephony, HITL…). Per-user OAuth with automatic refresh.
N+1→1 gateway: each tenant's own MCP servers are mounted as namespaced proxies on the same gateway — cache, circuit breaker and health PER server (one slow tenant server never degrades the others), hot-reload via Redis pub/sub.
1Password secrets: the agent handles secret://alias/field, resolved locally right before execution — default-deny, registrable-domain binding (SSO subdomains pass, bank.com.evil.com is blocked), result scrubbing under three encodings (raw, HTML, URL). ~19 dedicated tests.
Code execution: Claude Code in a separate Docker/K8s sandbox (4 GB / 2 CPU, ephemeral workspace, 7 lifecycle hooks, session resume).
Integrity-hashed audit: a per-row content_hash computed on the plaintext at write time — a contract pinned by a regression test — plus a SHA-256 root over id:hash pairs at export. Tampering is detectable by a third party, without any decryption key. Exportable PDF evidence packages (beta).
Round-trip PII: 100% on-prem GLiNER detection + regex (SIN, health-insurance numbers, cards…), anonymization before the model call and response re-identification via reverse-map — the external LLM never sees the real value, the user sees an intact answer.
Single chokepoint (BEFORE_BROADCAST hook) for AI consent (fail-closed) and the monthly budget (a real block — but an established voice call is never cut off; the block lands on the next turn). One crossing point covers web, Telegram, Teams and automations — a quota bolted onto the chat UI alone leaks through automations.
Direct SIP telephony (trunks, REGISTER, credential rotation without restart) — minutes at trunk price, not CPaaS price. And call_colleague: a text agent can phone a colleague, goal injected into the voice agent, result delivered back to the original room.
OpenAI-compatible API: POST /api/v1/chat/completions (streaming included) with your Luge key.
Security — the details that matter in a pentest
The WebSocket token travels in Sec-WebSocket-Protocol — specifically so it never shows up in access logs.
The local llama-server is locked down even on loopback: ephemeral port + a random 24-byte API key per process, passed via environment variable (not a world-readable cmdline); orphaned processes are tracked and killed.
Outbound webhooks: SSRF-validated registry (double resolution), X-Luge-Signature-256 HMAC signature, per-destination delivery ledger with a copy of the exact payload sent.
Measurable edge hygiene: zero panic!()/TODO in ~9,850 lines of Rust, 70 tests, clippy as a hard gate on 3 OSes.
What's not there yet
Windows system audio (WASAPI) — written, not wired yet. Local transcription on Windows — on the way. macOS and Linux are complete.
Edge inference is beta; local transcription is production.
Contributor-expertise weighting in memory recall exists in the formula… but equals 1.0 everywhere: the plumbing is waiting for its engine.
Improvement-loop fixes apply without an approval gate — a deliberate position: pre-mutation journal, bit-for-bit revert, blacklist of reverted fixes.
No ANN index on embeddings (a documented multi-tenant trade-off) — recall is exact, not approximate, and we watch the latency.
The platform code isn't open source. The orchestration foundation (RoomKit) is; the distribution repo with signatures is public. We'd rather say it ourselves.