mesalogo
d80d42b032
Main no longer opens desktop business SQLite. A single utility-process
storage owner holds the Assistant, Knowledge, attachment, document-result,
remote-binding and native model-ledger databases and runs whole domain
transactions; Main and Runtime callers use explicit async domain operations.
- Remove the Main readWithFallback path and the supervision worker's
separate business write connection; manifests are scanned read-only and
committed by the sole writer.
- Remote events are ACKed only after commit; process loss reports
unconfirmed writes instead of replaying them.
- Use native utilityProcess messaging with one in-flight bound (32) and
ordered waiting instead of capacity rejection and retries.
- Start the storage process before the window so startup overlaps.
- Keep short tool output in memory and store full output once in owned
storage; paging, branching, restart and deletion reuse that copy.
- Put required runtime launch files under goodbuddy-prefixed owned temp
directories and reclaim confirmed stale ones; stop writing duplicate
parsed.md.
- Add lint guards for SQLite ownership and architecture, concurrency
acceptance fixtures, and maintenance/architecture documentation.
Validation: typecheck and lint pass. Alternating baseline (edf019aa) vs
candidate runs show no regression on single-session, C01, C02, C04 and a
5-minute C06; Main p99 drops from ~21 ms to ~16.7 ms under concurrency.
Fresh portable copy (1.75M events) upgrade content unchanged; real-model
output paging passes with 4 external requests. Full npm test runs had
1-9 different timing-sensitive failures per run that pass when rerun.
Not covered: C03 review publication, C07 source body, app-level C05.
130 lines
8.1 KiB
Markdown
130 lines
8.1 KiB
Markdown
# Desktop File Storage Integration (S04)
|
|
|
|
Implemented domain operations, async consumers and production composition on
|
|
2026-10-05. The storage owner is registered by the foundation and the current Main
|
|
composition passes its file facades to ContextManager, document parsing and model
|
|
output. This does not claim that the full application has passed concurrent or
|
|
packaged acceptance.
|
|
The governing constraints are [performance principles](./performance-principles.md)
|
|
and [storage ownership](./desktop-storage-direction.md).
|
|
|
|
## Foundation API
|
|
|
|
`src/main/desktop-storage-files.ts` exports the complete integration surface:
|
|
|
|
- `DesktopFileStorageDomains`: intersect into `DesktopStorageDomains`. The domains
|
|
are `attachments` and `documentResults`.
|
|
- `attachmentStorageMethods` and `documentResultStorageMethods`: explicit allowed
|
|
operations. `attachments.reconcile` is an additional owner composite, with
|
|
arguments `[startup?, protectedIds?]`; its callback never crosses IPC.
|
|
- `openDesktopStorageFiles(userDataRoot, hasOwner, persistentResultLookup?)`:
|
|
construct once inside the existing utility process, after assistant and knowledge
|
|
databases open. Pass the existing userData root, not the assistant DB directory.
|
|
Add that root to foundation options and forward it in the client's open message.
|
|
- `DesktopStorageFilesOwner.dispatch(domain, method, args, signal)`: dispatch the
|
|
two domains here from the existing owner's dispatcher. It normalizes transported
|
|
original bytes, checks operation allowlists, and supplies cancellation to result
|
|
saves. Keep the existing transport admission and result limits.
|
|
- Before readiness, run `files.attachments.reconcile(hasOwner, true)`. The callback
|
|
uses the already-open assistant owner. It checks released references, marks
|
|
interrupted parsing, removes stale call-only output refs, and reclaims owned
|
|
orphan UUID directories. Do not run it concurrently with file producers.
|
|
- After draining admitted requests, `await files.close()` before closing assistant
|
|
and knowledge. Close drops call-only output refs, preserves history refs, removes
|
|
temporary document results, and closes the existing attachment database.
|
|
|
|
No new database, RPC transport, SQL endpoint, or process is introduced. Existing
|
|
attachment reference transactions and schema 1 remain in place. New relative paths
|
|
use slash separators; reads and reconciliation also accept released Windows paths.
|
|
|
|
## Main And Knowledge
|
|
|
|
After `storage.ready`, production composition uses `createDesktopStorageFiles(storage)` to obtain
|
|
`{ attachments, results, backingStore }`. Pass attachments to `ContextManager` and
|
|
results to `DocumentParsingService` and Knowledge's result-access dependency.
|
|
Do not also construct `ConversationAttachmentStorage` or `DocumentResultStorage`
|
|
in Main. Facades only call the existing storage client; importing them does not open
|
|
SQLite. Their result `close()` is intentionally inert: shared result shutdown belongs
|
|
to `DesktopStorageFilesOwner.close()`, after all producers drain.
|
|
|
|
Context operations that now require await: `storePastedImage`, `enrichRequest`,
|
|
`remove`, `serializeForQueue`, `restoreFromQueue`, `getDraft`, `hasImageInputs`,
|
|
`validateForSend`, `saveDraft`, `sendOriginal`, and `cancelUnavailableImport`.
|
|
Previously async import/reparse/image operations now await every storage operation.
|
|
Conversation/target validators may return Promises. Memory-only `clear` and
|
|
`activeContextIds` remain synchronous. `ContextManager.dispose()` cancels and drains
|
|
the active import before collecting unreferenced assets. Stop other context producers
|
|
and await their operations before disposing. Parsing-service disposal drains all
|
|
diagnostics, including those without an operation ID.
|
|
|
|
IPC must await attachment/result calls, including `results.original(id)` before
|
|
`shell.openPath`, and array predicates/maps after `get`, `draft`, `pendingParsing`,
|
|
and `getDraft`. Replace callback-based Main reconciliation with
|
|
`await attachments.reconcile(false, contextManager.activeContextIds())`.
|
|
|
|
Knowledge must accept `DocumentResultStorageAccess`, await `detach`, `original`,
|
|
and `isTemporaryResult`, and remove Main `setPersistentLookup` registration. Supply
|
|
that lookup to `openDesktopStorageFiles` instead: use owner-local
|
|
`knowledge.getDocumentByParsedResultId(id)` and the existing
|
|
`knowledge-assets/<libraryId>/<documentId>/<resultId>` directory and source location.
|
|
Keep Knowledge's publish/move/detach/release order. Result release never deletes an
|
|
external source original. No user-data directory migration is needed.
|
|
|
|
Project deletion supplies the project's conversation IDs captured before assistant
|
|
deletion to `attachments.deleteProject(conversationIds, protectedIds?)`. It removes
|
|
their refs in one attachment transaction, then collects only unreferenced assets.
|
|
Shared assets remain available to other conversations. Call this after the assistant
|
|
deletion commits; startup reconciliation covers an interrupted cleanup. Use
|
|
`deleteConversation` for single conversations and reconciliation for full clearing.
|
|
|
|
## Full Output
|
|
|
|
Pass the returned `backingStore` to each existing `PagedOutputStore`. It implements
|
|
`create/append/finish/read/release` directly in the existing attachment owner.
|
|
An output handle's UUID identifies its attachment row throughout capture, promotion,
|
|
history retention, restart, and deletion. Capture writes `original.txt` directly in
|
|
the owned attachment directory. Promotion renames that directory; no temporary full
|
|
copy or second backing is created. `request.json` contains only the descriptor.
|
|
Appends are bounded to 64 KiB; backing reads allow 32 KiB plus UTF-8 lookahead.
|
|
Small outputs remain in the existing bounded in-memory capture.
|
|
|
|
For history retention, await
|
|
`attachments.outputAdopt(callOwnerId, handle, conversationId, kind, referenceId)`
|
|
before the runtime releases its call reference. It returns the resource ID and adds
|
|
the reference without replacing the owner's existing attachments. Persist the
|
|
complete-output handle/reference in the owning event/message path; a preview alone
|
|
is not full output. Retain the call until required history adoption/persistence has
|
|
settled. Failed history publication uses existing release/reconciliation, not a
|
|
second durable operation log. Incomplete captures cannot be adopted.
|
|
|
|
`release` drops only the matching `output-call` ref. Persisted history can page after
|
|
call disposal or restart with
|
|
`attachments.outputRead(conversationId, handle, cursor, length)`, reading the same
|
|
bytes; it must use the owning conversation ID. History deletion uses the existing
|
|
message/queue/conversation reference cleanup. Runtime cancellation must await writer
|
|
abort/release. Startup removes stale call refs while retaining valid history refs.
|
|
The remote Agent path does not use this desktop backing and is unchanged.
|
|
|
|
## Validation Boundary
|
|
|
|
The focused suites exercise real SQLite/files through owner operations and async
|
|
facades, using structured-cloned arguments/results to cover Buffer-to-Uint8Array
|
|
conversion. They cover complete UTF-8 paging, one backing file, adoption, release,
|
|
abort, restart, project deletion with shared history, existing path separators,
|
|
temporary-result cleanup, failed adoption retry, async restore, and shutdown drain.
|
|
New result writes and copies omit `parsed.md`; old manifests and directories remain
|
|
readable. Result image writes settle sequentially before failure cleanup.
|
|
|
|
Local Windows validation: six focused suites passed (83 tests), covering this owner,
|
|
attachment references, ContextManager, parsing service, paged output, and HTTP OCR
|
|
integration. Scoped ESLint and diff whitespace checks passed. Node type checking
|
|
reported no diagnostics in S04 source/tests; the complete checkout still has pending
|
|
integration errors in Main/IPC and concurrent domain migrations.
|
|
|
|
These tests do not replace the foundation's utility-process tests or UI acceptance.
|
|
The real utility-process coverage for these domains is present in the focused
|
|
storage fixtures. Concurrent desktop acceptance and packaged validation remain
|
|
outstanding. No latency or throughput improvement is claimed from these focused
|
|
tests. Test resources live under `temp/goodbuddy-files-upgrade` and individual
|
|
fixtures remove their files.
|