goodbuddy/docs/architecture/desktop-storage-direction.md

Desktop Storage And Concurrent Execution

Decision and delivery

Updated 2026-10-05 against 2be57492. The user selected one complete architecture delivery for PERF-15/PERF-16 and the necessary performance guards. This document supersedes the earlier proposal to leave full migration for a later discussion. The integrated migration and acceptance are not complete. The implemented storage foundation, production composition and remaining acceptance work are recorded in Storage Foundation Integration. Local fixes in b90ff50f remain the measured starting point.

Deliver the integrated design in one candidate, with normal product workflows and concurrent acceptance passing together. Internal implementation dependencies do not create separate partial product deliveries. A worker-only prototype, an async facade over Main SQL, or a dual-write transition does not meet completion criteria.

This document owns architecture and scope. Performance principles own normative budgets and concurrency rules; concurrent acceptance owns reproducible workloads and evidence. The roadmap tracks status, and the local repair record preserves earlier measurements. Migration completion requires new acceptance evidence.

Requirements

ID	Required outcome
SA-01	Desktop Main does not open or query business SQLite, directly or through a helper.
SA-02	Complete domain transactions execute under one write owner per database.
SA-03	Multiple projects, conversations and subtasks progress without unrelated global waits.
SA-04	Requests, event buffers, result payloads and caches have bounded lifetimes and capacity.
SA-05	Persistence, ordering, cancellation, recovery and user interaction semantics are preserved.
SA-06	Startup, process failure and shutdown have one tested ownership path.
SA-07	Static guards and concurrent production-path tests enforce the architecture.
SA-08	GoodBuddy eliminates unnecessary intermediate disk writes and duplicate full outputs; necessary temporary resources have explicit ownership and cleanup.

Responsibilities

flowchart TB
  UI[Renderer: domain stores and visible views] <--> MAIN[Main: trusted IPC and lifecycle]
  MAIN <--> STORAGE[Storage utilityProcess: domain operations and write owners]
  STORAGE <--> READ[Existing bounded read workers]
  MAIN <--> RUNTIME[Runtime hosts and compute workers]
  MAIN <--> AGENT[Remote Agent]

• Renderer retains presentation, local interaction state and disposable visible-data caches. Domain selectors subscribe narrowly. Hidden views retain required state, not continuous full rendering or whole-history refreshes.
• Main retains windows, trusted sender/schema checks, OS APIs, credential decryption, network/runtime orchestration and process lifecycle. SQL, large result assembly, hashing and bulk attachment file work do not execute on its event loop.
• One desktop storage utility process owns the existing business database files and exposes typed asynchronous domain operations. Keep synchronous SQLite inside this dedicated environment where it simplifies transaction correctness.
• Reuse bounded read-only workers for expensive reads. They open only after upgrade, read committed WAL snapshots, and do not run initialization or recovery. Storage owns their lifecycle; a project or conversation does not create a new worker.
• Parsing, OCR, embedding and Runtime execution retain their existing hosts. Network requests and model execution must not run inside storage transactions or hold the storage write queue. This work does not relocate browser-dependent OCR engines.
• Remote Agent databases remain remote. Changes to desktop storage must preserve remote replay, controller ownership and commit-before-ACK.

One write owner means exclusive business-write authority, not necessarily one OS thread for all reads. Startup upgrades and existing maintenance/checkpoint workers must be explicitly owned and excluded from competing business writes. Keep their current bounded mechanisms where valid; do not create independent writers per domain.

Ownership inventory

Inventory records current owners; destinations below are targets, not implementation facts.

Surface	Current entry	Required destination
assistant.sqlite	src/main/assistant/assistant-database.ts, constructed in Main	Storage service: conversations, tasks/events, activity, notes metadata, channels, supervision and timing persistence
knowledge.sqlite	src/main/knowledge/knowledge-service.ts and knowledge-database.ts	Storage service: schema, writes and external metadata; existing workers for costly reads
conversation-attachments.sqlite and referenced files	src/main/conversation-attachment-storage.ts	Storage service: references, adoption, reconciliation and bounded file IO
Document-result persistent lookup and note bodies	document-result-storage.ts, magic-notes storage	Async domain access to associated files; preserve original directories and reference semantics
Local remote-runtime binding DB	src/main/remote-agent/managed-remote-execution-services.ts	Storage service: transactional binding transitions and uniqueness
Native terminal / DS Web model-call ledgers	native-terminal-client.ts, native-dsh-web-client.ts instantiate AgentModelCallLedger	Storage service manages desktop ledger files under the existing client lifecycle; Main retains model/credential routing
Read, checkpoint and upgrade connections	readonly-query-worker.ts, assistant-storage-worker.ts	Service-owned steady-state reads/checkpoints; exclusive upgrade execution before readiness
Supervision manifest writer	readonly-query-worker.ts supervision mode and SupervisionReviewStore	Reuse scanning and manifest logic, but route its business mutations through the same write owner as publication; remove independent writer authority
Remote Agent journal, owners, prompts and ledger	src/agent-daemon/daemon.ts, runtime-composition.ts	Existing remote ownership, unchanged by desktop connection migration
Encrypted JSON settings and transient runtime caches	Existing settings/runtime services	Keep existing appropriate owners; async settings do not move merely to match a diagram

An import scan must also find openPrivateSqliteDatabase and transitive constructors: an implementation under agent-daemon can still open SQLite when called in desktop Main. Do not weaken shared Agent code or relocate remote databases as a side effect.

Preserve existing database names, table formats, IDs, paths, scopes and durability settings. Process relocation alone does not require a schema bump. Necessary schema changes follow the migration guide.

Existing work and consolidation

This inventory was checked against production callers on 2026-10-05. Reuse means preserving working algorithms and contracts, not claiming their current transport, cancellation or close semantics already meet the target. Implement the replacement at the existing seam and remove superseded callers in the same delivery.

Existing mechanism	Reuse or extend	Retire or avoid
ReadonlyQueryReader, worker handlers	Reuse request correlation, named operations, errors, lazy lifecycle and reader handlers; adapt the outer utility transport and strengthen typed operations, readiness, byte bounds and settlement-aware close	No second equivalent storage RPC client or admission queue; remove production infrastructure fallback through readWithFallback, not legitimate owner-local transaction reads
Startup storage, upgrade worker	Preserve migration algorithms, progress, retry and cancellation; move SQL preflight into the storage-owned exclusive startup path	Remove Main SQL preflight and competing upgrade launch ownership; do not add a new migration framework
Checkpoint and supervision connections, review store	Reuse PASSIVE checkpoint and manifest scan/revision algorithms; make checkpoint exit awaited and manifest writes owner-controlled	No second maintenance service or independent business writer hidden in a read-worker entry
EmbeddingIndexRepository, binding store interface, activity repository	Reuse domain interfaces and transaction algorithms; adapt the desktop backing implementation and async callers	Do not add remote object proxies or duplicate repositories; Promise-returning methods are not proof of off-Main execution
RemoteEventBatcher	Extend existing batching with awaited persistence/flush and pending-plus-in-flight bounds; update recovery, background and interactive IPC consumers	No second durable event coalescer, receipt ledger or retry queue; no ACK or forwarding before commit
AgentModelCallLedger, terminal client, DS Web client	Preserve schema, claim/deduplication and uncertain outcomes; desktop access becomes awaited storage operations	Do not relocate remote Agent ledgers; remove desktop concrete synchronous opens and early directory deletion
Utility transport, Harness readiness, shutdown helpers	Reuse fork/entry/exit adapters and phased application shutdown; add storage readiness and drain at those seams	No new generic process supervisor, duplicate shutdown path or copied Harness streaming/version protocol
Runtime controller, selected runtimes, remote backend	Keep request ownership, session reuse and retirement; inspect remaining global open/prepare/close waits with healthy peers	No second Runtime pool or global execution manager; the prior route-cleanup fix does not prove every global wait is gone
Subagent scheduler, supervision model pool, review admission	Retain task slots, review single-flight and per-model-call pool as distinct existing scopes; extend capacity/settlement tests where needed	Do not merge ordinary chat into the supervision pool or create another task state machine/provider scheduler

The current ordinary reader removes a cancelled request from pending before worker settlement, and close() provides no exit barrier. Supervision waits longer and can substitute cancellation for a successful result. Extend these semantics to represent actual execution and committed outcomes; do not copy them unchanged into the storage client and call its capacity or shutdown complete.

Supervision's temporary scan table belongs to one SQLite connection. Retain the bounded scan, transfer only bounded manifest data to its write owner, and preserve initialization/resume invariants. Existing 200-row commits run in synchronous loops; they do not themselves yield an execution turn. Yield at safe existing batch boundaries without dividing final atomic publication or introducing a general scheduler.

Renderer consolidation stays within existing domains:

• Conversation store and persistence already own ordered deltas, save serialization, history requests and quit flushing. Extend those paths, not a second conversation-sync queue. Its current React-aware integration is not a reason to introduce another store library.
• Activity sync owns paging, pending writes and retry; task sync owns task/artifact refresh. Tighten projections and retention there. A resolved flush that caught an error and requeued data is not a durable success barrier; extend its explicit completion semantics where dependent reads or shutdown require persistence.
• Live messages, public event buffering, and terminal batching serve distinct display stages. Keep them separate from durable remote batching and avoid an additional universal event bus or a second coalescing delay.
• Retain domain cache owners: keep-alive, Markdown, channel status, and Supervisor scope reads. Preserve pinned-pane exceptions and scope/revision guards; strengthen bounds at their existing owners.

Runtime session behavior remains authoritative in the reuse design. Task timing and supervision model scheduling retain their existing semantics. Ending an elapsed-time segment on abort does not release an execution slot before cleanup; do not unify these clocks or restore historical timing scans/statistics workers during this migration.

Completion includes a removal inventory tied to the table above: old Main opens, external live-store factories, production sync fallbacks, obsolete transport callers and independent business writers have no production references. Remove only the superseded implementations, not valid in-owner synchronous algorithms, remote stores, released-data compatibility or historical test evidence. No permanent dual path or unused compatibility wrapper is part of the delivery.

Operation boundary

Reuse repository algorithms behind domain operations: save conversation changes, commit remote events, publish a review, adopt attachment references, or transition a runtime binding. Public renderer IPC can keep its current shape while Main awaits the storage client. No generic SQL endpoint or function serialization is introduced.

Existing transaction callbacks and returned live repository objects stay inside the owner. Replace their external use with typed inputs/results. In particular, readSnapshot, Knowledge publication callbacks and Supervisor store factories must not be transported or accept an asynchronous callback that outlives a sync transaction.

The channel extends the request/handler and utility-adapter seams identified above, correlates requests/results and carries named operations, serializable arguments, cancellation and readiness/failure. Reuse installed Electron transport patterns; do not add a general RPC dependency, persisted transport sequence, protocol negotiation or a service discovery layer for this bundled child process. Validate actual transport behavior for buffers, Maps and errors; use DTOs. Live classes, subscriptions and handles remain with their owners. Errors retain usable codes and causes without credentials or user content in diagnostic logs.

Queries return the requested projection. Lists use summaries/pages; source bodies, tool outputs and attachments load through detail paths. Large results must not be JSON-assembled in Main or copied through several caches. Existing source records are not truncated to satisfy a latency check. A read transaction ends before posting its reply so a slow consumer cannot keep a WAL snapshot open.

Concurrency and ordering

Keep ordering at the smallest required scope. A conversation's persisted sequence and a database's commits remain ordered; independent model waits, sessions and read requests do not share a global long-running lock. Runtime process reuse retains per-binding/session/operation identity; cancelling one request must not stop its peers.

Use existing finite concurrency and batch mechanisms before adding scheduling code. Apply capacity to queued bytes as well as operation count where payload size varies. When a queue fills, wait or report the existing actionable busy/error state; never silently discard user messages, tool results or acknowledged events. Obsolete read refreshes and display progress can coalesce; persisted evidence cannot. Preserve one bounded batch of work before yielding so other admitted producers can progress; change scheduling only where concurrent measurements demonstrate starvation.

Heavy reads use the read workers and leave necessary writes able to progress. UI/control requests must not wait behind model networking, import parsing or route cleanup. Control processing must not depend on acquiring another execution slot. This is a resource-boundary rule, not a mandate for a new global priority scheduler.

For long publications, compute immutable data before the write transaction when possible, then validate current database preconditions inside the transaction and commit the complete result/checkpoint atomically. Do not await models, filesystem callbacks or IPC while holding a write transaction. An atomic publication cannot be split into externally visible partial commits to improve timing. Worker relocation does not eliminate the single-writer bottleneck: acceptance measures queue wait and commit time, as well as Main responsiveness.

Persistence and cancellation

Operation	Required completion meaning
Cached UI preference/read	Synchronous memory access is allowed; cache ownership and invalidation are explicit
Existing coalesced local saves	Preserve current flush semantics and expose the existing flush barrier to dependent reads/exit
Remote event batch	Persist event/checkpoint under existing durability, then notify/ACK; enqueueing is not commit
Channel outbox / runtime ledger	Preserve duplicate-dispatch prevention and uncertain-outcome handling; no blind retry after lost replies
Review publication	Facts, result status and checkpoints commit together or roll back together

Abort removes queued work before dispatch. In-flight work reaches a defined safe point and reports its actual outcome; capacity is released only after settlement or confirmed host exit. Caller cancellation, operation termination and resource release are distinct observations, not three new persistent product states.

worker_threads SharedArrayBuffer cancellation must not be assumed to cross utilityProcess IPC. Reuse it only within verified thread boundaries. A message cannot interrupt synchronous SQL while the receiver is blocked. Keep SQL/transaction units bounded, check cancellation between safe units, and report a completed commit as committed even if cancellation arrived too late. Do not terminate a shared storage process to cancel one request or promise instantaneous rollback of a committed write.

Filesystem adoption and SQL references are not automatically one atomic transaction. Preserve existing staging, adoption and reconciliation order with async file APIs. Do not delete originals or release protected in-flight files early; do not introduce a cross-database transaction engine or recovery journal.

Lifecycle and failure

Main starts and supervises one storage process using existing utility-process patterns. Process spawn is not readiness: required migrations, connection opening and startup recovery must finish before storage-dependent operations are accepted. Show the existing startup/upgrade progress and retry UI. Render the application shell when safe without marking unavailable data as an empty successful result.

After process loss, reject affected pending requests with bounded diagnostics, stop admitting writes, confirm exit and close transport resources before replacement. The existing user retry/reopen lifecycle can re-establish storage readiness. Read-only work can be repeated when requested. For writes without a response, consult existing task/event/result identities and deduplication before claiming failure or retrying; model dispatch marked uncertain must not be sent again automatically. No Main SQL fallback, silent empty database, second durable receipt ledger or unbounded restart loop.

Shutdown stops producers and new admission, cancels unnecessary reads, drains required writes, closes readers/checkpoints/connections, and observes storage process exit. Extend the existing phased shutdown and deadline, including explicit storage failure reporting; existing settled-error helpers alone do not prove a successful drain. Native client ledger directories are removed only after their connection is closed. Upgrade/maintenance requires exclusive ownership before changing files, following the existing maintenance path; it cannot open an extra Main business connection.

Output and temporary resource ownership

This work is included in the same architecture delivery. The source investigation records current producers and persistence gaps. Scope is GoodBuddy-created material; third-party MCP/Runtime internals and user workspaces are not cleanup targets.

Decide content ownership before choosing a temporary directory. Ordinary requests, responses, concatenation and short outputs stay in bounded memory. Where an existing record or artifact already contains complete content, pass its reference and read pages from that owner. Do not write a temporary copy merely to cross IPC or call MCP.

Current tool events often contain only previews. Preserve full paging access by adapting the existing PagedOutputStore interface to the storage/attachment owner: output that must outlive bounded in-memory capture has one backing content object, with events carrying previews and its reference. Write directly to that backing object; do not spool to Temp and later copy the same bytes to another full-output store. Reuse existing file adoption/reference/release operations and paged reads, without a second output database, generic blob service or durable transport journal. Final answers, previews and full stdout/stderr remain distinct data.

The storage object's retention follows its consumer. Call-only output is released when its existing owner no longer needs paging; content referenced by persisted history is released through that history's existing reference cleanup. Being stored by the storage service does not make every intermediate output permanent. Keep small call-only output in memory and preserve existing model limits. Avoid adding whole-output copies to events or retaining all concurrent streams in an unbounded buffer. Size alone does not justify an additional temporary copy.

The replacement must specify the complete-content reference and byte cursor through process, search, subagent and output_read paths before removing the old file backend. Inspect remote mapping separately: it does not use the local process spool and cannot recover omitted bytes from already-pruned Agent events. Preserve supported remote content/ACK semantics; this work does not promise new full-history capture from third-party runtimes. Remove the old spool creation and synchronous Main paging only once the shared replacement preserves the existing full-output contract.

Necessary temporary material

External CLI configuration, plugin modules and atomic replacement/install staging may still require files. Reuse environment/in-memory configuration when already supported; otherwise keep the actual external contract. Atomic sibling files and installer staging stay on their required filesystem under their existing owner, not a forced central root.

For disposable runtime launch material, pass a root from application initialization through existing constructors; create it lazily:

<userData>/temp/goodbuddy-runtime/<appVersion>/<runId>/
  goodbuddy-runtime-launch/

Add purpose directories only for evidenced necessary files. Do not precreate process, search or subagent output trees. User-visible disposable roots retain goodbuddy- prefixes. Tests use <repository>/temp/goodbuddy-<task>-<runId>/; remote resources use GoodBuddy-owned prefixed roots on the Host. Do not modify global TEMP or relocate third-party histories/caches through an indiscriminate environment override.

The creator removes partial creations on failure. Existing service shutdown closes children/handles and then releases its resources. Startup, including the next upgrade, reclaims obsolete owned run directories after the existing single-instance check and verification that no owning child remains active. Version/run IDs identify candidates; they do not establish inactivity. Reuse process ownership checks, with only the minimal owner information needed to identify surviving children; do not create a lease service. Cleanup runs in bounded background work and cannot delay the first screen with a full filesystem scan. Failed deletion leaves the known directory available to retry.

Legacy OS-Temp prefixes require an explicit list of layouts, creator and inactivity evidence. Reclaim confirmed obsolete instances, preserve ambiguous/live directories, and report the remaining scope. Never delete arbitrary goodbuddy-*, .tmp or .partial paths by age. Existing installer/reference reconciliation retains its own roots and rules; no global garbage collector, cleanup ledger or user-data migration is introduced.

KISS constraints

• Keep Electron, TypeScript, SQLite and existing provider/runtime integrations.
• Add the one storage utility process, not processes per project/session/subtask.
• Reuse reader, checkpoint, parser, batching and lifecycle mechanisms where appropriate.
• Keep credentials/OS integration in Main; storage receives only required business data.
• Do not add general schedulers, distributed transactions, durable RPC journals, compatibility readers for unshipped intermediates, new approval gates or user modes.
• Synchronous cache reads and pure calculations need not become RPC calls.
• The runtime host, database driver and UI framework are not replaced as incidental work.

VS Code reference

VS Code demonstrates service separation, asynchronous execution, lazy activation, cached state and coordinated close. Its application/profile/workspace storage is owned by Electron Main, uses async @vscode/sqlite3, and batches cached key/value writes. GoodBuddy retains DatabaseSync in a dedicated execution environment because its relational history, graph queries and durable event acknowledgements differ. Do not copy full-state memory loading or empty-store recovery into business history.

Sources reviewed on 2026-10-05, upstream main; no same-machine benchmark is claimed:

• Connection ownership
• SQLite backend
• Cache and flush lifecycle
• Extension hosts

One-delivery completion

SA-01 through SA-08 must pass together under the acceptance matrix. All listed desktop owners, normal/background/error/cancel/exit callers, and native client ledger paths must use the final boundary. Remove the superseded Main openings, direct storage access and synchronous fallbacks, including helper-based bypasses. No feature-disable workaround or loss of user interaction counts as completion.

Run production UI-to-storage tests, existing full validators, old-data integrity and concurrent before/after measurements; validate affected desktop-to-Agent behavior on the shared Linux x64 Host. Native platform CI verifies process startup, SQLite and packaged entry/resource paths. Follow the release runbook for production builds and packaged-app checks, including measurement runs.

The acceptance record must report latency, throughput, backlog and resource cleanup, including failures. A responsive UI with stalled background commits is not success. No target implementation, new performance baseline or migration completion is claimed by this documentation update.