KIWI
Autonomous local AI agent and community of voices living in The Grove with private memory.
The Problem
Autonomous AI agents often suffer from leaky memory isolation, unpredictable side-effects on host systems, and lack of human-in-the-loop governance for high-stakes file operations and external device actions.
My Role
Architected and implemented the cognitive engine ("The Grove"), fail-closed host action safety policies, immutable exact-action plan ledgers, multi-device agent companion protocol, and unified multi-channel gateways.
Highlights
- Community of six voices living in The Grove with local memory running on the user's machine
- Local cognitive core ("The Grove") separating owner memory from credentialed client sessions
- Fail-closed security policy requiring immutable exact plans with rollback for all destructive actions
- DPAPI-secured Windows companion agent with local execution receipts and anti-re-execution guarantees
- Multi-modal interaction surface: Next.js Web UI, Voice HUD (:4321), and Telegram bot gateway with inline approvals
- Deterministic audit ledger tracking every state mutation, file movement, and memory association
Stack
Constraints
- Host safety is non-negotiable: no autonomous file deletion or batch reorganization without human-confirmed exact plans.
- Offline-first engine: core cognitive loops and memory databases must run purely locally without cloud dependencies.
- Multi-tenant memory isolation: administrative sessions and API token holders must never access the owner private mind context.
System Architecture
Key Trade-offs
The decisions worth defending — what I chose, what I turned down, and why.
File management execution model
Chose
Immutable two-phase exact-action plans with hash verification and dry-run preview
Rejected
Direct agent tool execution with broad shell permissions
Agents making speculative shell or file modifications risk catastrophic data loss; two-phase plans guarantee human oversight and rollback capability.
Engine-to-web communication architecture
Chose
Dedicated loopback HTTP/SSE gateway with bearer token authentication
Rejected
Monolithic single-process Next.js application
Decoupling the AI engine process from the Next.js frontend ensures background monitoring, telegram polling, and long-running skills survive web redeployments.
Device pairing security model
Chose
Windows DPAPI machine-bound tokens with polling architecture
Rejected
Inbound SSH or remote socket servers on client machines
Inbound listening ports expose clients to local network scans; outbound HTTPS polling behind DPAPI keeps remote companion devices secure.
What I'd Do Differently
An honest retrospective — the stuff I'd change with more time, more users, or a second pass.
- 1Implement automated regression tests for memory graph decay sweeps earlier in the development lifecycle.
- 2Consolidate SQLite schema migrations between better-sqlite3 and pg-schema under a single unified migration runner.
- 3Add fine-grained telemetry dashboards to observe cognitive latency across multi-turn reasoning chains.
Interactive Autonomous Agent Shell
Test live KIWI autonomous agent supervisor routines (agent status, grove ping, mcp tools, help) in the emulated shell below.
Technical Deep-Dive
Architecture, specifications, and implementation details.
P0 — State ownership contract
Revision 14: the Windows pilot tasks row gains Drizzle/PG schema parity and is now included in GDPR account deletion (device_tasks/user_devices/device_pairings, deleted by user ID). Fixing this uncovered and corrected an unrelated pre-existing defect: the GDPR delete route's transaction used an async callback, which the synchronous better-sqlite3 driver rejects outright — every account deletion request was failing with HTTP 500 across all tables, not just the new device ones, with no prior test coverage. Verification: 132 engine tests unchanged; 230 mind/web tests in 35 suites (+2); both typechecks passed. No production rollout. Details: Windows pilot.
Revision 13: crash recovery adds a new readable state to the contract. The owner's scoped thread memory is owned by the engine (ConversationManager, getTurns read-only copy), and the web uses it on GET /api/chat/history?channel=web&sessionId=... for admins (reconcileOwnerChat) to repair only the last message: if the last web message is a user with no assistant after it and the last completed engine round has the same user content, the missing reply is appended. The engine remains the canonical source of the completed round; the web SQLite store remains authoritative for displayed history. Best-effort (an engine error never blocks reading) and idempotent (at most one append). This does not close embedded personal persistence or consolidation of historical stores. Verification: 101 engine tests passed, one optional semantic skip; 222 mind/web tests in 33 suites; both typechecks passed. No production rollout. Details: chat, host actions.
Revision 12: Telegram update_id deduplication and atomic quota consumption are now durably recorded. The webhook records (chat_id, update_id) in telegram_webhook_updates (composite key; 200 on duplicates) and consumes the quota via a single conditional UPDATE (consumeMessageIfUnder); the engine persists the update_id offset in data/telegram-owner-offset.json, resuming from it after a restart — durable state for Telegram delivery, without double processing and without overshooting under concurrency. Verification: 97 engine tests passed, one optional semantic skip; 210 mind/web tests in 31 suites; both typechecks passed. No production rollout. Details: chat, host actions.
Revision 11: the chat routes (web and engine) now enforce body and message bounds. The web reads the body through web/src/lib/chat-body.ts with a 500,000-byte cap (streaming read), validates JSON (400 for a missing/invalid body or empty message) and rejects messages over 100,000 characters with 413 on both routes (/api/chat JSON and /api/chat/stream SSE). fileIds stays raw so resolveChatAttachments rejects malformed values (non-array, non-string ids). The engine (/api/chat/scoped) rejects non-string or over-cap messages with 413 without contacting the manager. Verification: 97 engine tests passed, one optional semantic skip; 205 mind/web tests in 30 suites; both typechecks passed. No production rollout.
Plan revision 13, 2026-09-09. Inspected behavior and migration boundaries; not a completed memory consolidation. Revision 13 added the owner chat crash recovery (read-only scoped thread from the engine + tail-only web reconciliation); revision 12 added durable state for Telegram delivery (update_id dedup + atomic quota + engine offset); revision 11 imposed the body/message bounds before any call to the brain or engine. Revision 10: ConversationManager serialized per-thread delivery (runSerialized), the engine exposed DELETE /api/chat/scoped (validating the owner-web:v1:<sha256> identity and waiting for the in-flight turn before forgetting the thread), the web, on deleting an admin browser session, asked the engine to forget the matching scoped thread without blocking on engine failure, and POST /api/documents/organize refused any non-dryRun. Verification (rev 11): 96 engine tests passed, one optional semantic skip; 200 mind/web tests in 29 suites; both typechecks passed.
Revision 8: chat transport continuity routes admin browser JSON and SSE through the same owner endpoint. An engine failure is explicit and never switches to the embedded account mind; the browser never automatically repeats a failed or partial request through JSON. SSE rejects Authorization headers, including alongside an admin cookie. The owner reply is buffered, not incremental model streaming. Current verification: 192 mind/web tests in 28 suites, web typecheck, isolated production build and Edge chat probe passed. Persistent personal state, internal engine fallback and explicit conversation boundaries remain open. No production deployment.
Revision 7: host action authorization now enforces tool scope in JSON/stream/offline paths, composes configured and caller restrictions, preserves the originating approval policy and prevents internal routine text from authorizing mutations. Known host changes require confirmation; previews are complete or refused. Direct Telegram rechecks policy. Autonomous dev launch, bulk document moves and disk reaping are blocked pending exact plans. Revision 7 validation: 90 engine tests passed, one optional semantic test skipped; 180 mind/web tests in 27 suites passed. P0 remains open for memory continuity, background automation/plugin paths and exact target migration. No production rollout.
| State | Current owner and storage | Boundary / remaining work |
|---|---|---|
| Web account identity | NextAuth session or validated account API key | currentUserIdFrom rejects cookie fallback whenever Authorization exists but is invalid; ISO download applies the same rule. Companion tokens cannot approve browser actions. Other session-only routes still need review. |
| Chat attachments | Owned chat_files, conversation and managed upload root | Transactional upload quota, bounded deletion and explicit expiry; unknown/legacy files preserved. Expiry does not erase text already copied into memory/history. Lifecycle. |
| Personal web mind | web/src/lib/kiwi/state.ts, process-local Map keyed by user ID | Attitude, relationship, active persona and the last 200 memories are volatile. On restart only the last 40 owner-filtered chat_messages are reloaded. Conversation boundaries are currently mixed within an account. |
| Personal observations | nature_observations, queried by user_id | Interpretations must not be represented as verified facts; provenance/correction and restart evaluations remain P4 work. |
| Legacy host mind | src/kiwi.ts, src/kelvin.ts, engine memory stores and conversation manager | Owned by the operator, not by all website users. Direct Telegram allowlisted users administer this same mind/host. |
| Admin web continuity | Scoped endpoint hashes validated account/conversation; recent history and approvals remain separate | Operator semantic memory, persona and long-term state remain shared. Unscoped legacy route retains fallback. Crash recovery (revision 13): the engine's scoped thread is a read-only reconciliation source; the web SQLite store remains authoritative for display. Chat contract. |
| Web Telegram | telegram_links and owner-filtered chat_sessions/chat_messages | Private chat identity must equal sender; pairing is atomic and expiring. Legacy sessions are reused only for the same owner; new IDs hash owner + Telegram chat. Ambiguous active links fail closed. |
| Document assistant | Server-resolved account sandbox; request-local document context | Chat is now stateless, grounded only in that request's documents, without owner memory/tools/status. History continuity needs an owner-scoped implementation. |
| Windows pilot tasks | device_pairings, user_devices, device_tasks in web SQLite, with Drizzle/PG schema parity (revision 14) | Account/device/action digest and expiry are authoritative on the server; the agent rechecks them and maintains a local execution receipt. No dependency on either brain's RAM state. Cleared on GDPR account deletion (revision 14). |
#Enforced now
/api/kiwi/status and /api/kiwi/memory, like JSON chat, reserve legacy owner-engine access for an admin browser request without Authorization. Other requests receive their personal embedded state. Every memory response — source: embedded (personal) or source: engine (owner) — carries only memories/total. [Updated rev 69] The tiered STM/MTM/LTM subsystem was deleted (it never persisted anything); the owner branch now reads the engine's real flat memory table instead of always returning empty stm/mtm/ltm arrays. See docs/adr/0009-three-tier-memory.md.
Conversation confirmations are keyed by original channel identity, even when history is shared. They contain a copied action, expire after five minutes and recheck policy before execution. Web Telegram linking requires a matching private sender and does not regenerate expired pairing codes from remote messages.
The document skill gateway rejects arbitrary host tools, userId overrides and actual organization moves. The engine clones the request's document context instead of changing the global directory. The document assistant no longer calls the shared conversation manager or owner fallback. This intentionally removes tool execution and multi-turn memory from that path until a scoped replacement is available.
Auto-resolve checks expiry at decision/execution and limits model tools to diagnostics; a broad “fix” approval no longer authorizes arbitrary model-selected modifications. Legacy coding/OpenCode and other direct host mutations remain outside the new device contract.
#Next migration and acceptance
The target is one persistent logical personal mind per authenticated account, with explicit conversation IDs, provenance and versioned state. Devices execute approved tasks; they do not become new authoritative memory stores. Shared rooms may receive only opted-in, evaluated, sanitized lessons, never account snapshots.
Before migration: enumerate remaining host mutations, scope conversations and the internal engine fallback, implement transactional account persistence and deletion/export behavior, then compare restart, concurrent request and cross-account tests. Preserve mind-library/ and all source PDFs. No PDFs or historical conversations were deleted or imported by this change.
Evidence: P0 audit, Windows pilot, checkpoint. Tests establish the listed boundaries, not a complete project security assessment.