goodbuddy/docs/architecture/desktop-storage-foundation.md

Desktop Storage Foundation Integration

Implemented 2026-10-05. This is the storage foundation for the desktop storage direction. The production Main composition now opens one DesktopStorageClient and supplies its typed facades to the migrated domains. Full application acceptance is still incomplete. Do not open the existing Main databases alongside the owner.

Client API

Import DesktopStorageClient from src/main/desktop-storage-client.ts in Main. Construct it after Electron readiness and the application's single-instance check. The existing database directories must exist. Preserve the current database paths.

const storage = new DesktopStorageClient({
  assistantPath,
  knowledgePath,
  defaultRootPath,
  userDataPath,
  onProgress: progress => publishExistingUpgradeProgress(progress),
  onChanged: domain => publishExistingDomainNotification(domain),
  onFailure: error => reportStorageFailure(error)
})
await storage.ready

await storage.call('assistant', 'saveLocalConversations', [changes])
const conversation = await storage.call(
  'assistant', 'getConversation', [conversationId], { signal }
)
const results = await storage.call('knowledge', 'search', [searchOptions], { signal })

// After stopping producers and awaiting their persistence flushes:
await storage.close()

call(domain, method, argumentTuple, { signal? }) infers argument and result types from the explicit allowlists in desktop-storage-contracts.ts. All results are Promises. Pass cancellation as the transport option, not inside a DTO. Functions, signals and live class instances are rejected before posting. Maps and Uint8Arrays are supported; Node Buffers arrive as Uint8Arrays across Electron IPC.

userDataPath is the existing application user-data root, not a database directory inferred from a filename. ready means upgrade worker exit, database initialization, attachment startup reconciliation and runtime-binding owner initialization completed. Calls before readiness fail without queueing. retry() retains the same public client and facades. A live failed startup retries in that process; a failed/exited utility is replaced only after exit confirmation and old-request settlement. The replacement gets fresh request correlation, never the old writes. The confirmed upgrade decision is acknowledged into Main memory before upgrade execution, retained across startup loss, and cleared when that upgrade completes. Wire retry() to the existing retry UI. There is no automatic restart or replay of unconfirmed writes.

The defaults locate desktop-storage-entry.js, readonly-query-worker.js and assistant-storage-worker.js as packaged siblings. entryPath, readerWorkerPath and upgradeWorkerPath overrides support isolated tests and explicit composition. The new entry is included in electron.vite.config.ts.

Owner-Side Operations

Existing call shape	Storage operation
Assistant/Knowledge ordinary database methods	call('assistant' / 'knowledge', method, args) using the explicit allowlists
SSH project create/update	assistant.createSshProject / assistant.updateSshProject accept Omit<SshProjectWrite, 'assertCurrent'>; Main checks the live host/runtime before dispatch, and the owner retains cancellation and project-revision checks inside the existing transaction
activityHistory() live repository	Existing getActivityHistoryPage, getActivityHistorySummary, updateActivityHistory, replaceActivityHistory, clearActivityHistory, reconcileActivityHistory operations
supervisionReviewStore()	call('review', method, args); initialization/resume execute on the write owner, not the old independent writer
supervisionStories()	call('stories', method, args)
supervisionExperiences()	call('experiences', method, args)
supervisionSuggestions()	call('suggestions', method, args); resolution uses assistant.resolveSupervisionSuggestion
Knowledge externalStore	call('external', method, args)
readSnapshot(callback) for review context	call('assistant', 'supervisionContext', [request]); summary/background share one reader snapshot
Knowledge publishDocument(..., { afterChunksInserted })	call('knowledge', 'publishDocument', [input, chunks, { graph, embeddingReplacement, embeddingError }])
Knowledge graph-only replacement	call('knowledge', 'replaceDocumentGraph', [library, document, graph]); reuses the existing evidence replacement transaction
Attachment references and full-output backing	call('attachments', method, args); includes owner-local reconcile(startup?, protectedIds?)
Temporary and persisted document results	call('documentResults', method, args)
Runtime bindings and native model-call ledgers	call('runtime', method, args)

Repository instances never leave the owner. The publication graph DTO is the existing GraphExtractionResult. The owner creates the graph callback and performs entity identity resolution, evidence insertion, chunk/embedding publication and pruning within KnowledgeDatabase.publishDocument's existing transaction. Replace the consumer's publication callback with this DTO; remove its duplicate graph publication path when migrating that consumer. Extraction/model work remains outside storage. Publication and replacement share storeExtractedGraph from desktop-storage-knowledge-operations.ts. Callback-bearing replaceEvidenceForDocument is excluded from the transport allowlist. File adoption/release and task-status orchestration remain outside this SQLite transaction.

After readiness, createDesktopStorageFiles(storage) supplies attachments, results and output backing; createDesktopRuntimeStorageAdapters((domain, method, args) => storage.call(domain, method, args)) supplies runtime bindings and ledger access. The facade caller types now share the core DTO types, so these calls need no casts. StorageData preserves the existing recursive JsonValue as-is rather than recursively rebuilding it; runtime validation still rejects callbacks and handles.

The owner supplies both file callbacks. Attachment reconciliation checks assistant.hasAttachmentOwner. Persistent document lookup reads knowledge.getDocumentByParsedResultId and resolves the existing <userDataPath>/knowledge-assets/<libraryId>/<documentId>/<resultId> directory and source location. Main must not register another persistent lookup on the results facade. The existing onProgress, onChanged and onFailure APIs are unchanged, including createAssistantStorageOnChanged's four notification mappings.

Keep these composites intact during integration:

• Remote task/conversation batches use the existing appendRemote*Batch or appendRemote*Once methods. Await commit before forwarding or ACK. Never translate a batch into independently acknowledged per-event writes.
• Review publication uses assistant.saveSupervisionResult, which owns result facts and checkpoint commits. commitCheckpoints is deliberately not an external review operation. No callback-based transaction API is exposed.
• Story/experience apply and review batch save retain their existing repository transactions. A model request must finish before calling these operations.
• Additional multi-statement snapshots need a named owner/reader handler; do not recreate readSnapshot as several awaited Main calls.

Bounds And Lifecycle

The utility adapter retains ReadonlyQueryReader request correlation, errors and admission. Its existing request limit is 32, with 32 MiB of argument admission credit. This is no longer a content-size restriction: one oversized complete input uses the entire credit budget exclusively. Smaller concurrent inputs still share the budget; saturation returns STORAGE_CAPACITY without dropping work.

desktop-storage-wire.ts transfers complete values on the same channel in frames of at most 256 KiB accounted data. Each direction sends one frame and waits for its acknowledgement before constructing the next. Strings are split without assembling a whole JSON/serialized copy; binary frames copy only their slice and fill the receiver's final buffer. The receiver exposes a value only after its last frame. These frame acknowledgements provide transport backpressure, not durable write ACKs. Cancellation bypasses the frame queue and is retained for an incompletely received request until dispatch can reject it safely.

Known complete-note/history/result reads share one materialization/transfer slot, so queued requests do not build many huge reply objects while another reply drains. Writes and bounded queries retain ordinary admission. A complete user value itself still needs memory proportional to its size; the frame budget is not a claim that total process memory or every supported history fits in 32 MiB. There is no result size rejection or truncation, and no STORAGE_RESULT_CAPACITY outcome after a commit.

Renderer conversation persistence now targets 8 MiB batches as well as the existing 100-conversation bound. A single oversized conversation stays complete and uses the framed transport. Quit flushing drains all remaining batches and propagates a failed write. Oversized single histories no longer fail solely because of the transport cap. STORAGE_UNCONFIRMED means the process exited without confirming a pending operation; reconcile using existing domain identities, not blind retries.

Queued cancellation is checked before dispatch. In-flight reader cancellation uses the existing thread SharedArrayBuffer inside the owner; that buffer is never sent over utility-process IPC. Synchronous commits cannot be interrupted by an incoming message, so a completed commit returns success even when cancellation was requested too late. Request capacity remains occupied until settlement.

close() stops client admission, drains admitted requests, awaits file-owner cleanup, closes runtime ledgers/bindings and read workers, awaits the existing PASSIVE checkpoint worker's exit, closes both database owners and observes utility-process exit. An exit without the close acknowledgement rejects close. Apply the application's existing shutdown deadline to the whole operation; the transport separately reports an unconfirmed exit after its 10-second exit wait. Startup failure follows the same release barriers. A failed close attempts the remaining independent closes, reports STORAGE_CLOSE_FAILED, and exits unsuccessfully; it does not acknowledge a clean drain or permit a second owner to open in that host.

Remaining Integration And Acceptance

Supervision And Heartbeat Callers

createSupervisionDomainPorts(storage) in src/main/assistant/supervision-domain-ports.ts supplies Main-side RPC facades:

const ports = createSupervisionDomainPorts(storage)
const supervisor = createProductionSupervisorService(ports.supervision, getSettings, resolveRuntime, pool, persistUsage)
const phrase = createProductionSuggestionPhraser(ports.supervision, getSettings, resolveRuntime, pool, persistUsage)
const heartbeat = new HeartbeatService(ports.heartbeat, actions)
await deriveSuggestions(ports.suggestions, phrase, input)

The production factory's other arguments are unchanged. If provided, persistUsage must return the persistence Promise. All heartbeat methods, including configuration CRUD and history, now return Promises. Review repository factories may return a Promise of the typed facade. The facades contain closures only; no database or live repository is transported. initializeSupervisionReview and resumeSupervisionReview route to review.initialize / review.resume, with signals in transport options. The caller does not invoke the old independent supervision writer.

story-assignment-service.ts and experience-extraction-service.ts are the sole story/experience model orchestrators, including their prompts, output schemas and model-facing types. Production, tests and live scripts import those services. The repository files retain their SQL methods and transactions, with type-only imports for validated model output. supervision-suggester.ts awaits candidate selection and suggestion publication. The heartbeat live script also awaits plan creation before using its ID.

Callers await leaf/navigation writes, final publication, task usage/status, heartbeat projections, story/experience results and error bookkeeping. Cancellation aborts model work immediately, but review admission remains occupied until the cancel write and active execution settle. A late rejected cancel does not overturn a confirmed publication. Failure to persist a suggestion failure is propagated; a model phrasing failure whose status was saved preserves the completed review.

Validation includes unchanged behavioral assertions in the existing review/story/ experience suites, delayed-commit tests, and supervision-storage.test.ts: a real Electron storage utility process runs heartbeat, two review leaves, navigation, stories, experiences, suggestion publication, usage, a no-change heartbeat and cancellation using a local deterministic Runtime. This does not validate final Main/IPC composition or concurrent latency budgets.

On 2026-10-05, after removing the duplicate orchestrators, the targeted supervision/ heartbeat run passed 215 tests across 21 files, including the real utility-process test. Scoped ESLint and Node type checking passed.

The bounded scripts/supervision-storage-live.cjs <runtime-settings.json> probe used the current supervisor profile (gpt-6-astra, OpenAI Responses) and production utility storage with three synthetic calibration messages. Exactly four actual HTTP requests completed, all HTTP 200: review, story assignment, experience extraction and suggestion phrasing. It committed one leaf covering all three sources (589 characters), assigned three events to one story, created one experience and published two suggestions. The unchanged review made zero additional requests. Persisted usage was four calls, 3,495 input and 1,251 output tokens. Two initial preflight attempts sent zero requests; the probe was corrected to install the isolated Electron encryption state before readiness. Current settings remained byte-identical; credentials were not logged, and isolated databases, encryption state and bundles were removed. This is functional evidence, not concurrent performance acceptance.

Integrated Owner

The foundation registers assistant, knowledge, external metadata, supervision, attachments, document results and runtime operations without Main SQL fallback. The production composition in src/main/index.ts constructs the Assistant, Knowledge, file and Runtime facades after storage readiness and passes them to the relevant Main services. Remote Agent database ownership is unchanged.

The remaining work is product-level acceptance: complete C01-C07/A01-A10 coverage, packaged cross-platform checks, and removal or verification of any caller that is outside the migrated desktop domains. Focused domain tests do not establish those results.

The S05 runtime-domain owner, Main adapters, required composition wiring and focused validation are documented in Runtime Storage Integration.

Supervision now uses ReadonlyQueryReader.reviewManifest(runId). The existing read-only assistant worker scans once under a read snapshot into connection-local temporary metadata, ends the snapshot, and returns at most 200 metadata rows per page. It holds at most one live scan manifest, rejects a competing scan explicitly, and releases that temporary data in the operation's settlement path. It never opens a business-write connection.

The sole write owner clears/rebuilds disposable manifest rows in bounded batches, commits each insertion batch atomically, and yields between commits. Cancellation and persisted pause/cancel status are checked between batches. initializing remains set after interruption; clearing it and deleted-source cleanup share the final transaction. Resume rebuilds partial manifests, or validates existing frozen revisions in read-worker pages of at most 200 rows, yielding between pages. A failed scan never falls back to synchronous owner scanning. Final result/checkpoint publication remains the existing atomic operation.

The old writable supervision worker mode, connection opener and separate reader field were removed. AssistantDatabase's direct initialization/resume methods use the same bounded implementation; in-memory owner-local use retains its synchronous algorithm because a separate read connection cannot share that database. Other non-worker operations still execute synchronously inside storage; no latency improvement or full concurrent performance acceptance is claimed.

The current focused validation passes 70 tests across eleven suites: core storage, file/runtime domains, assistant facade, read workers, bounded review-store algorithms, existing supervision worker regressions, startup, wire backpressure, renderer batching and real supervision/storage orchestration. Real Electron coverage includes graph replacement rollback, SSH revision conflicts, owner-side persistent document lookup after reopen, full-output retention and crash reconciliation, ledger recovery, runtime startup failure/retry and pending file-save drain before cleanup. Review tests cover 200-row bounds, concurrent owner writes, frozen scan revisions, cancellation, failed-batch rollback, reader loss, marker recovery, deleted sources and final marker/cleanup atomicity.

Real Electron large-value tests save an explicitly typed 16 MiB MP4 payload, read a complete two-entry note, and round-trip all 25,600,230 characters of an 80-message history. Hashes verify complete video and message bodies. The measured accounted frame high-water marks were 248,332 bytes in Main and 252,578 bytes in storage, below 262,144 bytes; argument credit peaked at 33,554,432 bytes. Main RSS peaked at 426,049,536 bytes including fixture inputs, complete results and application memory. This is queue-bound evidence, not a whole-process memory or latency budget claim. The real process tests also kill an in-progress write transfer and confirm no replay after retry, and verify startup decision retention/clearing across process loss.

Full Node, Agent and renderer typechecking passes, as does scoped ESLint. Temporary checks and databases use temp/goodbuddy-storage-foundation; tests remove their generated bundles/data. The earlier foundation production-build check predates this domain integration. Packaged cross-platform and full UI-to-storage concurrent acceptance remain outstanding.

Reader Ownership Cleanup

The obsolete readWithFallback export, database-owned readonly-reader fields and checkpoint lifecycle methods, and the database async wrappers were removed. Heavy reads are dispatched by DesktopStorageOwner; owner-local synchronous reads remain available inside transactions and focused database tests. Read-worker and checkpoint tests now create explicit ReadonlyQueryReader/Worker owners and await their close. The three perf harnesses create explicit owner-local reader/checkpoint resources under the repository temporary root, report executionBoundary: owner-local, and close those resources before deleting their copies. No production or test source reference remains to the removed compatibility APIs.

The cleanup validation adds termination barriers and failed-frame poisoning: a reader cannot reopen capacity while its prior worker termination is pending, a transport posting failure cannot feed later values into a partial decoder, and close waits for wire drain and owner settlement. The focused cleanup run passes 122 tests; Node and renderer type checks and scoped lint pass.