KIWI
Autonomous local AI agent and community of voices living in The Grove with private memory.
The Problem
Autonomous AI agents often suffer from leaky memory isolation, unpredictable side-effects on host systems, and lack of human-in-the-loop governance for high-stakes file operations and external device actions.
My Role
Architected and implemented the cognitive engine ("The Grove"), fail-closed host action safety policies, immutable exact-action plan ledgers, multi-device agent companion protocol, and unified multi-channel gateways.
Highlights
- Community of six voices living in The Grove with local memory running on the user's machine
- Local cognitive core ("The Grove") separating owner memory from credentialed client sessions
- Fail-closed security policy requiring immutable exact plans with rollback for all destructive actions
- DPAPI-secured Windows companion agent with local execution receipts and anti-re-execution guarantees
- Multi-modal interaction surface: Next.js Web UI, Voice HUD (:4321), and Telegram bot gateway with inline approvals
- Deterministic audit ledger tracking every state mutation, file movement, and memory association
Stack
Constraints
- Host safety is non-negotiable: no autonomous file deletion or batch reorganization without human-confirmed exact plans.
- Offline-first engine: core cognitive loops and memory databases must run purely locally without cloud dependencies.
- Multi-tenant memory isolation: administrative sessions and API token holders must never access the owner private mind context.
System Architecture
Key Trade-offs
The decisions worth defending — what I chose, what I turned down, and why.
File management execution model
Chose
Immutable two-phase exact-action plans with hash verification and dry-run preview
Rejected
Direct agent tool execution with broad shell permissions
Agents making speculative shell or file modifications risk catastrophic data loss; two-phase plans guarantee human oversight and rollback capability.
Engine-to-web communication architecture
Chose
Dedicated loopback HTTP/SSE gateway with bearer token authentication
Rejected
Monolithic single-process Next.js application
Decoupling the AI engine process from the Next.js frontend ensures background monitoring, telegram polling, and long-running skills survive web redeployments.
Device pairing security model
Chose
Windows DPAPI machine-bound tokens with polling architecture
Rejected
Inbound SSH or remote socket servers on client machines
Inbound listening ports expose clients to local network scans; outbound HTTPS polling behind DPAPI keeps remote companion devices secure.
What I'd Do Differently
An honest retrospective — the stuff I'd change with more time, more users, or a second pass.
- 1Implement automated regression tests for memory graph decay sweeps earlier in the development lifecycle.
- 2Consolidate SQLite schema migrations between better-sqlite3 and pg-schema under a single unified migration runner.
- 3Add fine-grained telemetry dashboards to observe cognitive latency across multi-turn reasoning chains.
Interactive Autonomous Agent Shell
Test live KIWI autonomous agent supervisor routines (agent status, grove ping, mcp tools, help) in the emulated shell below.
Technical Deep-Dive
Architecture, specifications, and implementation details.
P4 — Owner-memory isolation and retraction correctness
Revision 27, 2026-09-09. First real P4 slice. Per the plan's acceptance line — "retained claims are traceable and correctable, users remain isolated, and changes can be evaluated against a baseline" — an independent Opus planning pass scoped what's actually buildable now versus what's greenfield, and found two of the four sub-claims were each real but incomplete: a security property CHECKPOINT already claimed closed, and a correction/deletion path that worked for the dashboard but leaked back through two other retrieval paths. Local code and automated verification; no production rollout, no engine restart.
#Owner-memory isolation, closed at HEAD for the first time
CHECKPOINT revision 17 states: "GET /api/kiwi/knows is engine-backed only for an admin browser without authorization; other accounts get embedded per-user payloads." That was true only in the working tree. HEAD's committed web/src/app/api/kiwi/{status,memory,knows}/route.ts all gated the owner-engine branch on isAdminUser() alone — which resolves only the NextAuth session, never the Authorization header (web/src/lib/access.ts's isAdminUser() vs. currentUserIdFrom(), which correctly prioritizes an API key over a session). A request carrying a valid admin session cookie alongside a Bearer kiwi_ak_… machine token — realistic whenever the admin's own browser also holds an agent-key-authenticated session, e.g. via the CLI or a device agent — would read the owner's private engine memory through what's meant to be a machine-scoped, per-account path. This is exactly the bypass currentUserIdFrom already guards against everywhere else (/api/devices applies the identical rule).
Fixed by resolving identity and checking the header before the admin branch is attempted at all — status.ts previously tried engineFetch first and only resolved identity in the catch block, a fail-open ordering, not merely a missing check. A second, independent instance of the same gap: knows/route.ts's DELETE/PATCH (retract/edit a memory) checked isAdminUser() alone even in the working tree — a machine credential could mutate the owner's memory through a path it could not even read. Fixed identically.
#Retraction correctness — three real leaks closed
retractMemoryById (src/mind/database/database.ts, already committed, no pending diff) zeroes importance and stamps metadata.retracted = true — but three of its four sibling read paths never excluded retracted rows:
getMemoriesWithEmbeddings— semantic search scoressimilarity*0.7 + importance*0.2 + decay*0.1; a zeroed-importance row still clears typical recall thresholds on similarity alone. A "forgotten" fact kept reaching the model even though it had vanished from the dashboard.getLinkedMemories— the association graph kept resurfacing a retracted memory as a live one's "this reminds me of…" neighbor.decayMemories's hard-delete sweep —retractMemoryById's own docstring promises a kept, stamped row (an audit trail, not erasure), but the sweep deletes any row withimportance < 0.3, which a retracted row always has, the moment itsdecay_scorealso drifts below0.01. Left alone, a retracted memory would be permanently destroyed by the daily tick within roughly three weeks.
All three fixed with the minimal filter (NOT_RETRACTED, or the equivalent in-JS metadata check) already used consistently by the file's other five read functions.
#Security / correctness checklist
- Authorization symmetry. Read and write paths on owner memory now apply the identical rule (
req.headers.has('authorization') || !(await isAdminUser())) — previouslyGETandDELETE/PATCHdisagreed. - Fail-closed ordering. Identity resolves before the admin branch is ever attempted, so a missing/invalid credential 401s before either brain is touched.
- No new surface. Reused
engineFetch,currentUserIdFrom,isAdminUser— no new endpoints, no schema change for this half of the slice. - Retraction stays one-way and auditable. All three fixes exclude retracted rows from retrieval; none of them touch
retractMemoryByIditself, so the kept, stamped row (the audit trail) is unaffected — only which paths can still surface it to the user or the model.
#File-by-file
web/src/app/api/kiwi/status/route.ts,memory/route.ts,knows/route.ts— hot (large pre-existing uncommitted files, part of the same deliberately-deferred P0-era bucket as prior revisions' Telegram work). Built from HEAD's actual committed baseline in an isolated scratch copy, not the working tree, and staged viagit hash-object/git update-index— confirmed viagit diff --cachedto be exactly this fix, nothing from the surrounding pending content.src/mind/database/database.ts— clean (tracked, no pending diff at the time of this fix) — the three retraction-leak fixes, ordinary commit.tests/userMemory.test.ts— clean, extended with 3 new cases.web/tests/kiwi-memory-ownership.test.ts— new (7 cases).
#Deliberate non-goals for this slice
- Per-fact source/page provenance. No storage layer anywhere — engine
memories, web's embeddednature_observations, or the knowledge-searchChunktype — carries a document/page reference through to a stored memory today. Designing that schema across two unrelated stores, before any real requirement narrows the shape, would be guessing rather than building. - An eval harness ("changes can be evaluated against a baseline"). Nothing exists to compare against — no fixture set, no baseline run. Building one before the retraction leaks above were fixed would have baselined the buggy behavior; this is the natural next slice, not this one.
src/memory/*(STM/MTM/LTM tiers, MemCon, DiffCoT, commitment, firewalls). Real, deliberate design — but unreachable at runtime:ConversationManageronly activates the tiered path for aMemoryLifecycleinstance, andsrc/kelvin.tsalways constructs aMemoryStoreAdapterinstead, so nothing is ever written to it. Several further defects were found while reading it (a promotion step that clears 100% of short-term memory instead of the intended half, three independentMemoryStoreinstances silently overwriting one shared file, long-term memory permanently unrankable due to a hardcoded score, aforgetskill that reports success without deleting anything) — reported, not fixed, since the subsystem's fate (repair-and-wire, or delete) is an architectural decision for the operator, not a side effect of a memory-ownership fix.- The SQLite tier tables (
stm_entries/mtm_nodes/ltm_entries) sitting in the same pending bucket — real schema, but nothing writes to them; committing the tables without writers would be inventing state. - Unifying the three memory systems (engine tiered, engine flat SQLite, web embedded per-account) — a genuine architectural decision needing its own design pass.
#Verification
npm run typecheck and npm run typecheck:web: both green. Engine: 145 tests (144 passed + 1 optional semantic skip; +3). Web/mind: 41 suites / 271 tests (+7). npm run verify:imports: 496 files, 0 problems. Isolated clone (git clone --local --no-hardlinks + fresh npm ci + npx tsc --noEmit): clean except the pre-existing, unrelated stripe/index.ts/studio.ts migration gap documented in revision 26 — unchanged by this commit. [RESOLVED at rev 36] stripe/studio were deleted outright and every real importer migrated; typecheck:web has been clean on a fresh checkout since. No production rollout.
#Remaining P4 work
Per-fact provenance; the eval-against-baseline harness; the fate of — decided at revision 68 (delete) and done at revision 69, see src/memory/*'s dead tiered subsystemdocs/adr/0009-three-tier-memory.md's amendment; unifying the three memory systems; the pre-existing stripe/studio typecheck gap (revision 26, still open) — resolved at rev 36.
Sources: web/src/app/api/kiwi/{status,memory,knows}/route.ts, src/mind/database/database.ts, web/src/lib/access.ts, tests/userMemory.test.ts, web/tests/kiwi-memory-ownership.test.ts, checkpoint.