KIWI
Autonomous local AI agent and community of voices living in The Grove with private memory.
The Problem
Autonomous AI agents often suffer from leaky memory isolation, unpredictable side-effects on host systems, and lack of human-in-the-loop governance for high-stakes file operations and external device actions.
My Role
Architected and implemented the cognitive engine ("The Grove"), fail-closed host action safety policies, immutable exact-action plan ledgers, multi-device agent companion protocol, and unified multi-channel gateways.
Highlights
- Community of six voices living in The Grove with local memory running on the user's machine
- Local cognitive core ("The Grove") separating owner memory from credentialed client sessions
- Fail-closed security policy requiring immutable exact plans with rollback for all destructive actions
- DPAPI-secured Windows companion agent with local execution receipts and anti-re-execution guarantees
- Multi-modal interaction surface: Next.js Web UI, Voice HUD (:4321), and Telegram bot gateway with inline approvals
- Deterministic audit ledger tracking every state mutation, file movement, and memory association
Stack
Constraints
- Host safety is non-negotiable: no autonomous file deletion or batch reorganization without human-confirmed exact plans.
- Offline-first engine: core cognitive loops and memory databases must run purely locally without cloud dependencies.
- Multi-tenant memory isolation: administrative sessions and API token holders must never access the owner private mind context.
System Architecture
Key Trade-offs
The decisions worth defending — what I chose, what I turned down, and why.
File management execution model
Chose
Immutable two-phase exact-action plans with hash verification and dry-run preview
Rejected
Direct agent tool execution with broad shell permissions
Agents making speculative shell or file modifications risk catastrophic data loss; two-phase plans guarantee human oversight and rollback capability.
Engine-to-web communication architecture
Chose
Dedicated loopback HTTP/SSE gateway with bearer token authentication
Rejected
Monolithic single-process Next.js application
Decoupling the AI engine process from the Next.js frontend ensures background monitoring, telegram polling, and long-running skills survive web redeployments.
Device pairing security model
Chose
Windows DPAPI machine-bound tokens with polling architecture
Rejected
Inbound SSH or remote socket servers on client machines
Inbound listening ports expose clients to local network scans; outbound HTTPS polling behind DPAPI keeps remote companion devices secure.
What I'd Do Differently
An honest retrospective — the stuff I'd change with more time, more users, or a second pass.
- 1Implement automated regression tests for memory graph decay sweeps earlier in the development lifecycle.
- 2Consolidate SQLite schema migrations between better-sqlite3 and pg-schema under a single unified migration runner.
- 3Add fine-grained telemetry dashboards to observe cognitive latency across multi-turn reasoning chains.
Interactive Autonomous Agent Shell
Test live KIWI autonomous agent supervisor routines (agent status, grove ping, mcp tools, help) in the emulated shell below.
Technical Deep-Dive
Architecture, specifications, and implementation details.
P3 — Telegram device selection and file-edit approval
Revision 26, 2026-09-09. Chosen slice, on explicit request to proceed with P3 despite two prior planning passes (revisions 22 and 25) deferring it as "too big for one slice" — a third Opus planning pass found that assessment overstated the gap: DeviceControl.propose/decide (P1) already implement the approve/reject contract the plan asks for; what was actually missing was only the chat transport (callback_query/inline-keyboard handling, absent from the webhook entirely). Local code and automated verification; KIWI_DEVICES_ENABLED stays off by default, no production rollout.
#What this delivers, against the plan's literal acceptance line
"Acceptance: a Telegram request executes only on the linked selected PC after its required approval, and results match the web/TUI task record.
/deviceslists a user's paired PCs (online/offline, same 15slast_seen_atwindow the dashboard uses) with an inline keyboard; tapping one persists it as that chat's selected device (newtelegram_links.selected_device_idcolumn) and is confirmed by editing the message in place./write <path>then the exact new file content, starting on the next line, proposes awrite_filetask on the selected device — the sameDeviceControl.proposethe web dashboard's manual form calls, shown with ✅ Approve / ❌ Reject buttons.- Approving calls
DeviceControl.decide, then edits the message to a static confirmation with a 🚫 Cancel button, live until the device claims the task; rejecting callsdecide(approve:false); cancelling (a real gap found while building this, below) calls the newDeviceControl.cancel. /tasksreads back the last 5 rows from the exact sameDeviceControl.listcall the dashboard renders from — the acceptance line's "results match the web/TUI task record" by construction, not by a separate, parallel implementation.- Only ever creates a new file (
expectedSha256: null) — Telegram has no way to prove a file's current SHA-256, andsrc/devices/local-executor.tsenforces this with an atomiclinkSync(EEXISTif the file appeared), not merely by not offering the option.
#Security checklist
- Authorization.
callback_queryupdates carry nochat.type/self-identity guard of their own upstream, unlike the existing message path —handleDeviceCallbackUpdatere-deriveschat.type === 'private' && chat.id === from.idfrom scratch before touching anything, so only the linked account's own private chat can ever act on its own buttons. Group chats are excluded on both the send side (link.chatIdis only ever written from inside that same guard) and the receive side. - Replay. callback presses use the same
(chat_id, update_id)dedup mechanism the message path already relies on —update_idis strictly increasing per bot across every update kind, so no cross-kind collision is possible. (Implemented as its own small local check against the sharedtelegram_webhook_updatestable rather than importing the message path's version — see "Committed narrowly" below for why.) - Stale/foreign buttons. Every button's
callback_datacarries the first 16 hex characters of the proposal's digest;decide/cancelre-check the task's live digest still starts with that prefix before acting, so a button rendered for a since-changed or already-decided proposal cannot silently act on whatever the task became. - Rate limiting.
handleDeviceCommand/handleDeviceCallbackUpdatecallDeviceControldirectly (server-side, same process — the dashboard's own precedent for this isweb/src/app/api/admin/tick/route.tscallingDeviceControl.reapdirectly), which bypasses/api/devices's ownrateLimit()calls entirely. Found and fixed while building this: Telegram now applies the same per-account keys and ceilings as the HTTP route (device-tasks,device-tasks-decision,device-tasks-cancel) — without this, Telegram would have had no proposal/decision throttling at all, on top of the dashboard's own budget. - Revoked/expired devices. Fails safe with no new logic:
/devicesonly ever offers non-revoked devices; a stale selection surfacesDeviceControl.propose's existing 404; the task's 5-minute TTL (unchanged by approval) anddecide's own state/expiry checks are the same ones the dashboard already relies on.
#A real gap in DeviceControl found and fixed while building this
The plan's P3 acceptance line asks for "approve/reject/cancel". DeviceControl.decide only ever accepted awaiting_approval — there was no way to abort a task already approved but not yet claimed by a device, only revoke() (which kills the whole device, not just one task). Added cancel(userId, id, digest): aborts awaiting_approval or approved, refuses running or any terminal state. Exposed at POST /api/devices/tasks/cancel (same shape as tasks/decision) and from Telegram's post-approval button — this closes the gap for both interfaces, not just the new one.
A second, independent bug found in the same file: propose() counted pending tasks against the 20-per-device cap and then inserted the new row outside a transaction, unlike every other method in the class. Two concurrent proposals arriving at the cap could both pass the check before either inserted. Wrapped in db.transaction(...) like the rest.
#Committed narrowly on purpose
web/src/lib/telegram.ts and the webhook route are themselves large, pre-existing, uncommitted files — P0-era Telegram work (dedup, quota, file handling, preferences) left uncommitted by deliberate standing choice (see CLAUDE.md's Blocked/next list, which names "Telegram" explicitly). All new P3 logic lives in a new, self-contained module, web/src/lib/telegram-devices.ts, precisely so the two hot files only need a handful of small, isolated hunks (one line changed, four small Bot-API helpers appended, one type extension, one new branch, one new call) — built from HEAD's actual committed baseline in an isolated scratch copy, not the working tree, and staged with git hash-object/git update-index. Confirmed via git diff --cached that the two diffs are exactly the P3-related lines (99 total) and nothing from that ~500-line pending bucket.
One consequence: telegram-devices.ts deliberately does not import the still-uncommitted isTelegramUpdateDuplicate — it has its own few-line local dedup check against the same, already-committed (since revision 12) telegram_webhook_updates table, so this module has no dependency on that pending function's commit status.
Two more bugs surfaced while reading that pending bucket for this work, fixed in the working tree and verified via tests, but intentionally left uncommitted along with the rest of it:
getOrCreateTelegramConversationlooked up an existing conversation by the legacytg_<chatId>id format, but always inserted using a hashed id — the "reuse an existing conversation" branch could never match a real row, silently dead since whenever it was written.- The bare-
/starthandler sent a reply usinggetLinkByChatId(chatId)?.token ?? ''— for any never-linked chat (the common case for a bare/start) this is a guaranteed-to-fail send with an empty token, and for an already-linked owner it sent the wrong, confusing "waiting for its owner" message.
#Post-commit verification: an isolated clone found two more, unrelated, real regressions
Following the same discipline as revision 23, after committing this slice a fresh git clone --local --no-hardlinks + npm ci + npx tsc --noEmit was run against the new HEAD. Neither of the two P3 commits themselves produced any error — but the fresh checkout still failed to typecheck, for reasons that turned out to predate this revision:
web/src/app/api/user/data/route.ts(committed revision 24) importsdeleteAllSessionsfrom@/lib/chat-history, which had never itself been committed — chat-history.ts's committed version stopped atdeleteSession. This session's own revision-24 verification checked only the working tree (which already had the pending function) — the same blind spot the wholeverify:importseffort exists to close, except this dependency wasn't a relative import, so that guard didn't catch it.web/src/app/api/admin/tick/route.ts(committed earlier this session) importserrMessagefrom@/lib/errors, a small, dependency-free untracked utility.
Both fixed and re-verified in the same isolated clone (git fetch + re-run tsc --noEmit): the only remaining fresh-checkout errors are in web/src/lib/stripe/index.ts and web/src/lib/studio.ts, both already deleted in the working tree as part of an in-progress, deliberately-uncommitted Stripe→LemonSqueezy/persona-system migration — not touched here, and not part of P3.
This means npm run typecheck:web currently fails on a genuinely fresh checkout of master, independent of anything in this revision — worth the operator's attention before the next CI run or clean-machine checkout, since stripe/index.ts/studio.ts are still imported by HEAD's own committed webhook route (the old persona system), so simply deleting them would break the currently-committed webhook, not fix it. The real fix is committing more of the in-progress persona/billing migration together — out of scope for this revision by the same standing choice that deferred the rest of the Telegram bucket.
[RESOLVED at rev 36] Stripe/studio were deleted outright (payments run on LemonSqueezy) and every real importer was migrated to the new owner-chat/chat-attachments/chat-body system in the same commit. npm run typecheck:web has been clean on a fresh checkout since. The paragraph above is left as written at the time, not rewritten.
#Deliberate non-goals for this slice
- Voice. No speech-to-text exists anywhere in the web app.
- NLP-mediated task drafting ("chat naturally and Kiwi proposes the edit"). No interface has this today — the web dashboard is a manual form (device dropdown, path field, content textarea).
/writeis the honest structured peer of that form, not a shortcut below it; inventing untested NLP that emitswrite_fileactions blind would be the unsafe move here. - Content beyond Telegram's own ~4096-character message limit. The protocol's 64 KiB cap is unchanged; the transport is the limit. A cheap, documented follow-up: accept a document attachment whose caption is
/write <path>, reusing the file-handling functions already intelegram.ts. - A stateful multi-turn proposal flow ("send the path, then send the content as a separate message"). Deliberately avoided — it would need its own pending-input state machine and its own TTL. One message, one proposal.
- Proactive push of task results.
/tasksreads live state instead. Pushing would need to know which interface proposed a task (nothing tracks that today, across Telegram vs. the web dashboard) to avoid leaking a web-originated task's result into Telegram — a real gap, spec'd for a follow-up (a smalltask origintable plus a fire-and-forget hook inPOST /api/devices/agent/result), not guessed at here. - Dashboard parity for
cancel. TheDeviceControl.cancel/POST /api/devices/tasks/cancelcapability exists at the platform level and ships in Telegram, but the web dashboard's owndevices-panel.tsxwas left untouched — it is itself an untracked, pre-existing P1 file, and adding to it now would mean committing its other unrelated pending content too.
#Verification
npm run typecheck and npm run typecheck:web: both green. Engine: 142 tests (141 passed + one optional semantic skip; unchanged). Web/mind: 40 suites / 264 tests (+15: 12 new in web/tests/telegram-devices.test.ts, 2 new in device-control.test.ts for cancel, 1 new in device-api.test.ts for the HTTP tasks/cancel wiring). npm run verify:imports: 490 files, zero problems, zero undeclared packages. Isolated clone + fresh npm ci + npx tsc --noEmit: clean except the pre-existing, unrelated stripe/studio migration gap documented above. No production rollout.
#Remaining P3 work
Voice; NLP-mediated proposal drafting; content larger than one Telegram message; proactive result push (spec'd above); dashboard cancel button; two-account real-device acceptance (still blocked on a second physical Windows machine, same as P1); the pre-existing stripe/studio typecheck gap this revision surfaced but did not fix — resolved at rev 36 (see above).
Sources: web/src/lib/telegram-devices.ts, web/src/lib/telegram.ts, web/src/app/api/telegram/webhook/route.ts, web/src/lib/device-control.ts, web/src/app/api/devices/[[...path]]/route.ts, web/src/lib/db/{sqlite-schema,pg-schema,index}.ts, P1 device pilot, checkpoint.