From 4c7193e1eac04ae8fa89980bb1739ff6609baf29 Mon Sep 17 00:00:00 2001 From: mesalogo Date: Sat, 26 Sep 2026 12:42:08 +0800 Subject: [PATCH] fix: avoid history reclamation during structure-only upgrades MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ordinary database schema upgrades previously ran VACUUM on free pages and showed misleading history-cleanup progress. Separate structure upgrades from required legacy conversions, retaining reclamation intent only for retries within the same startup. Preserve released data without new schema versions or persistent migration markers; Agent and normal writes remain unchanged. Add real SQLite and Worker regressions, bilingual operation-specific startup copy, and migration documentation. Synchronize native Electron drag input with actual resize state without weakening layout assertions. Validation: 47 focused migration tests passed; typecheck, lint and diff checks passed. Final full suite: 5137 passed, 74 skipped, 2 unrelated Electron failures. Both failing Electron tests passed an isolated rerun; full CI remains unverified. Release note: 修复普通数据库结构升级再次整理历史和回收空间的问题；仅在需要转换旧数据时执行相关处理，启动页准确显示当前升级操作。 --- FEATURES.md | 4 +- FEATURES.zh-CN.md | 2 +- docs/development/database-migrations.md | 10 +- .../execution-history-storage.md | 13 ++- docs/features/assistant-workbar/progress.md | 25 +++++ docs/features/magic-notes/progress.md | 9 +- docs/features/magic-notes/technical-design.md | 16 ++-- docs/quality/recurring-defects.md | 23 +++++ src/main/assistant-storage-startup.test.ts | 77 +++++++++++++++ src/main/assistant-storage-startup.ts | 18 ++-- src/main/assistant-storage-worker.ts | 9 +- .../assistant/activity-history-io.test.ts | 3 +- src/main/assistant/assistant-database.ts | 17 ++-- .../assistant-storage-upgrade.test.ts | 53 ++++++++++ .../assistant/assistant-storage-upgrade.ts | 40 +++++--- .../magic-notes/magic-note-storage.test.ts | 10 +- src/renderer/src/StorageUpgrade.test.tsx | 31 ++++++ src/renderer/src/StorageUpgrade.tsx | 31 ++++-- src/shared/assistant-storage-contracts.ts | 2 +- tests/assistant-storage-worker.test.ts | 96 +++++++++++++++++++ tests/magic-notes-layout.electron.test.ts | 13 ++- 21 files changed, 441 insertions(+), 61 deletions(-) create mode 100644 src/main/assistant-storage-startup.test.ts create mode 100644 tests/assistant-storage-worker.test.ts diff --git a/FEATURES.md b/FEATURES.md index 59811594..8571dfd6 100644 --- a/FEATURES.md +++ b/FEATURES.md @@ -194,7 +194,9 @@ records are listed separately and do not introduce another feature status. reopen the upgraded database. The schema-35 repair released in 0.13.2 also handles repeated tool blocks written after upgrading to 0.13.1, without deleting events. Terminal tasks release their write caches; duplicate remote replay does not - rebuild a released cache. See + rebuild a released cache. Structure-only upgrades do not reconvert history or + reclaim ordinary free pages; the startup page distinguishes structure updates, + legacy conversion, and actual space reclamation. See [storage and upgrade behavior](./docs/features/assistant-workbar/execution-history-storage.md). - [x] **Direct model Runtime**: Supports question answering, knowledge diff --git a/FEATURES.zh-CN.md b/FEATURES.zh-CN.md index effea326..a5734c0e 100644 --- a/FEATURES.zh-CN.md +++ b/FEATURES.zh-CN.md @@ -41,7 +41,7 @@ ### Agent Runtime 与模型连接 -- [x] **子任务历史去重与空间回收**：本机与托管 SSH 子任务事件只存变化，不再反复保存完整进度。首次升级自动转换旧记录并回收空间，显示进度且支持退出重试；保留聊天、执行详情、结果和远程事件去重。已随 0.13.2 发布的 schema 35 补修还会整理升级到 0.13.1 后重复写入的工具块，不删除事件；终态及时释放写入缓存，远程重复重放不重建已释放的缓存。旧客户端不能打开升级后的数据库，具体规则见[执行记录存储与升级回收](./docs/features/assistant-workbar/execution-history-storage.md)。 +- [x] **子任务历史去重与空间回收**：本机与托管 SSH 子任务事件只存变化，不再反复保存完整进度。首次升级自动转换旧记录并回收空间，显示进度且支持退出重试；保留聊天、执行详情、结果和远程事件去重。已随 0.13.2 发布的 schema 35 补修还会整理升级到 0.13.1 后重复写入的工具块，不删除事件；终态及时释放写入缓存，远程重复重放不重建已释放的缓存。普通结构升级不重新转换历史或回收正常空闲页，启动页区分结构更新、旧内容转换与实际空间回收。旧客户端不能打开升级后的数据库，具体规则见[执行记录存储与升级回收](./docs/features/assistant-workbar/execution-history-storage.md)。 - [x] **直连模型 Runtime**：支持问答、知识总结、受控工具执行、图像生成，以及通过支持 OpenAI 兼容图片编辑接口的服务商进行参考图编辑。连续请求自动复用最近一次成功生成的图片和文字历史，已保存会话重开后仍可继续修改，本轮显式附图优先；上游明确不支持编辑时继续按文字生成，仅在回复底部提示未使用参考图。编辑使用 multipart 上传图片，结果仍只接受经过校验的内联图片，不下载服务商返回的图片 URL。详见[图片生成与连续修改](./docs/features/image-generation/README.md)。 - [x] **当前会话图片工具**：为图片模型开启会话调用后，支持工具调用的聊天模型可在 Execute 中生成图片、修改上传图片或历史成果，无需另开会话或手动切换图片模型。直连文本模型、本地 OpenCode、Continue、DeepSeek Harness 和托管远程 OpenCode 复用统一服务，Runtime 分配由模型设置派生；图片请求可能产生费用，编辑需服务商支持。详见[实现与验证记录](./docs/features/conversation-media-generation/progress.md)。 diff --git a/docs/development/database-migrations.md b/docs/development/database-migrations.md index c36d351f..d4f787a8 100644 --- a/docs/development/database-migrations.md +++ b/docs/development/database-migrations.md @@ -18,6 +18,11 @@ - 先修新增数据路径，再修历史数据。历史转换与新写入尽量使用同一套转换函数。 - 普通 schema 变更使用既有 `PRAGMA user_version` 和 SQLite 事务； 已升级 schema 不允许旧客户端强行打开，不通过回写版本号支持降级。 +- 将结构升级、旧内容转换和空间回收分别判断。结构版本落后不代表需要整理历史； + 普通更新产生的空闲页也不是旧数据证据。只有明确的旧内容转换才允许检查空闲页并 + 执行 `VACUUM`，后续新增结构迁移不自动继承全库回收。启动页按实际操作显示状态。 +- 同一次启动的重试可在内存中保留已确认的转换／回收决定；退出后重新根据数据库 + schema 和未转换内容判断，不为仅剩空闲页增加持久化标记或重复启动回收。 - 大数据转换不得同步阻塞 Electron Main 或 Renderer。放到工作线程，在业务连接 打开前处理，并使用应用的共享字体、主题与页面组件显示进度和可执行的失败提示。 - 根据实测设置有界读取批次与缓存，批次提交后再报告进度。保持 SQLite 事务与 @@ -40,6 +45,9 @@ 对象比较必须覆盖真实生产形状和 JSON 往返，尤其是可选字段值为 `undefined` 时与字段缺省 的差异。再次修复已发布迁移产生的数据时，提高目标 schema，并覆盖已升级用户的入口； 不能只处理尚未迁移的旧库。 +仅修改升级调度、不改变存储格式时不增加 schema。回归必须覆盖已完成历史转换的库 +在后续结构升级时仍有空闲页，确认不重新转换或 `VACUUM`，并验证真实 Worker、 +同次启动重试和下一次启动的区别。 ### 真实旧库 @@ -71,5 +79,5 @@ 表示，并回收空间。具体规则见 [执行记录存储与升级回收](../features/assistant-workbar/execution-history-storage.md)， 数据保留与真实旧库验证见[工作栏进度](../features/assistant-workbar/progress.md)。 -- 当前未发布源码：schema 35 同时整理旧快照和 schema 34 的重复工具块，工具字段按 +- Desktop 0.13.2：schema 35 同时整理旧快照和 schema 34 的重复工具块，工具字段按 JSON 可保存的值比较，不因 `undefined` 被省略而误判变化。 diff --git a/docs/features/assistant-workbar/execution-history-storage.md b/docs/features/assistant-workbar/execution-history-storage.md index cf31a2df..f1f8983f 100644 --- a/docs/features/assistant-workbar/execution-history-storage.md +++ b/docs/features/assistant-workbar/execution-history-storage.md @@ -13,6 +13,8 @@ 处理速度；处理时间取决于历史数据量与磁盘速度，不承诺固定几分钟。用户可以退出， 下次启动重新检查并继续转换未完成的数据；完成后回收空闲页并进入应用。失败时留在 该页提供重试与退出，磁盘空间不足会明确说明，不删除聊天来腾出空间。 +以上转换和回收只针对仍需转换的旧格式数据；普通结构升级使用独立状态，不宣称正在 +清理子任务历史，也不会仅因数据库有空闲页而执行全库回收。 ## 存储规则 @@ -94,7 +96,10 @@ Renderer 合并时独立接收 Main 的置顶属性，即使本地正文更新 启动检查只根据旧 schema 或仍未转换的笔记正文决定是否进入升级流程；正常写入、更新 与删除产生的 SQLite 空闲页不是旧格式证据。迁移后新增数据使用当前格式，不因正常使用 -再次迁移。同一次启动已进入的迁移流程可重试压缩；若正文已转换、仅压缩中断，退出后 +再次迁移。进入升级流程后仍分别判断结构、笔记转换与空间回收，不把旧 schema 当作 +回收许可。回收只用于实际存在旧子任务事件（schema < 35）、旧活动记录（schema < 44） +或旧笔记正文的转换；例如 schema 45／46 升到当前结构不会自动转换历史或 `VACUUM`。 +同一次启动在内存中保留这一决定，只有原本需要回收的流程可重试压缩；若正文已转换、仅压缩中断，退出后 重开不单凭空闲页重新进入迁移，剩余页由 SQLite 后续写入复用。不新增持久化迁移标记。 桌面在打开业务数据库之前启动独立 Node 工作线程，使用同一套编码和还原函数转换旧库。 @@ -108,8 +113,8 @@ Renderer 合并时独立接收 Main 的置顶属性，即使本地正文更新 在转换前复制整份膨胀内容。回收仍需要额外可用磁盘空间，失败保留已提交转换并允许重试。 子任务整理对应 schema 35；schema 34 用户也会执行该整理，成功后后续启动直接跳过。 -继续复用 schema 34 的子任务事件部分索引和同一差量表示。当前业务初始化随后在事务中 -升级为 schema 36，新增 `pinned INTEGER NOT NULL DEFAULT 0 CHECK(pinned IN (0, 1))`， +继续复用 schema 34 的子任务事件部分索引和同一差量表示。业务初始化随后在事务中 +执行后续结构迁移，其中 schema 36 新增 `pinned INTEGER NOT NULL DEFAULT 0 CHECK(pinned IN (0, 1))`， 已有会话默认未置顶，保留原消息与时间戳；失败回滚本次字段迁移。 低于该 schema 的旧版客户端不能重新打开已经升级的数据库。需要回退客户端时，必须使用 升级前备份，不要修改 `user_version` 强行绕过版本检查。 @@ -120,6 +125,8 @@ Renderer 合并时独立接收 Main 的置顶属性，即使本地正文更新 - 工具 fixture 必须含生产对象会携带的 `undefined` 可选字段，并经过实际 JSON 写入／重读。 同时验证旧完整事件和 schema 34 差量事件的整理、混合重试及无变化记录不被重写。 - 验证远程重复事件、冲突事件，以及消息更新失败时事件和过程缓存的事务回滚。 +- 覆盖已完成历史转换、但正常聊天更新产生空闲页的后续结构升级：无转换进度、无 + `VACUUM`，会话、笔记、任务事件与完整性不变；验证真实 Worker 和同次启动重试。 - 在真实旧库的已核验备份副本上运行 portable 的正常启动入口，核对事件逐条等价、 其他用户表内容、数据库完整性和实际文件大小，不能只统计空闲页或只测试内存 fixture。 - 同时验证真实本地 Runtime 和共享 Linux Host 的桌面持久化路径。真实场景必须让主代理 diff --git a/docs/features/assistant-workbar/progress.md b/docs/features/assistant-workbar/progress.md index 33547fdc..43bd89ec 100644 --- a/docs/features/assistant-workbar/progress.md +++ b/docs/features/assistant-workbar/progress.md @@ -1,5 +1,30 @@ # 工作栏实现与验证进度 +## 2026-09-26 结构升级与历史回收分离（未发布） + +- schema 45／46 的回归先复现普通结构升级执行 `VACUUM`，修复后不转换历史、 + 不回收正常空闲页，保留聊天、笔记、事件及 SQLite 完整性。 +- 启动决定在内存中分别保留笔记转换与回收需求，Worker 重试复用原决定； + 不增加 schema 或持久标记，不修改正常写入。规则见 + [旧库升级](./execution-history-storage.md#旧库升级)。 +- 真实 Node Worker 验证结构升级、旧笔记转换、取消与重试、文件大小和内容保留； + 启动调度回归验证允许／禁止回收的两种重试，中英文页面区分实际操作。 + 六个定向测试文件 **47 项通过**，类型检查、lint 和 `git diff --check` 通过。 +- 完整 `npm test`：**5,138 通过、74 跳过、1 失败**。未修改的 + `tests/magic-notes-layout.electron.test.ts` 原生拖拽断言失败：首次索引列保持 168， + 单独复跑则 AI 列保持 280，均未达到拖动后的预期宽度。该 fixture 直接加载 + Magic Notes 和模拟 API，不运行本次存储升级路径。随后修正测试的原生输入同步： + 等待焦点、拖拽建立、宽度更新及结束，并明确左键按下状态，未弱化断言或改变产品 + 布局；独立复跑及随后完整测试中的该用例通过。 +- 最终完整重跑：**5,137 通过、74 跳过、2 失败**，迁移相关及布局拖拽均通过。 + `inline-help.electron.test.ts` 超时，`overlay-layering.electron.test.ts` 子进程退出码 + 为 null；这两个未修改的用例随后一起独立复跑 **2 项通过**，未提高超时或修改断言。 + 最终 typecheck、lint、`git diff --check` 均通过。独立复跑不替代完整测试结果， + 不宣称本次全量或 CI 全绿。 +- 本机与托管 SSH 历史共用 Desktop 数据库升级路径；Agent、Runtime、模型桥和 + 远程协议未变，不需要部署 Host 补丁。本次未调用模型，未改写用户数据库， + 尚无本次修复的 CI 或发布验证。 + ## 2026-09-21 重复历史迁移补修（Desktop 0.13.9 候选） - 0.13.8 的检查仍在已有笔记时把 `freelist_count > 0` 当作迁移依据；正常聊天更新 diff --git a/docs/features/magic-notes/progress.md b/docs/features/magic-notes/progress.md index a1270788..3295196e 100644 --- a/docs/features/magic-notes/progress.md +++ b/docs/features/magic-notes/progress.md @@ -467,8 +467,13 @@ Electron cases with substituted model output; live-provider checks remain opt-in migration and backup requirements are defined in the [storage contract](./technical-design.md#file-storage-and-writes). - The startup worker converts one entry per transaction and verifies hydrated - readback before clearing the legacy payload. Reclamation checks the freelist - on every attempt, including retries after conversion has finished. Committed + readback before clearing the legacy payload. The original implementation checked + the freelist on every attempt. This was narrowed by the September 21 startup + fix and the September 26 structure-upgrade fix: only a required legacy + conversion enables reclamation, with its decision retained for same-startup + retries, not later launches. Current rules are in the + [migration contract](./technical-design.md#schema-38-migration). + Committed saves/deletes/resets remain successful when manifest or GC cleanup fails; failures are logged and cleanup is retried on later access or reopen. - Storage implementation handoff reports 135/135 focused tests passed across diff --git a/docs/features/magic-notes/technical-design.md b/docs/features/magic-notes/technical-design.md index e0844de1..7a022b50 100644 --- a/docs/features/magic-notes/technical-design.md +++ b/docs/features/magic-notes/technical-design.md @@ -175,12 +175,16 @@ Already converted entries are reconciled on retry. Matching body revisions do not require rehydrating binary payloads. Direct `AssistantDatabase.initialize` callers and tests also run storage repair. -Space reclamation is separate from conversion. Every worker attempt checks -`PRAGMA freelist_count` and runs checkpoint/VACUUM when free pages remain. -Cancellation after the last converted entry or a VACUUM failure therefore -allows reclamation to retry on the next startup, even with no legacy rows left. -The schema marker alone does not establish that conversion and reclamation -finished. General migration requirements are in the +Space reclamation is separate from structure upgrades and conversion. Startup +detects legacy note payloads and retains the conversion/reclamation decision in +memory for that startup's worker retries. Only a required legacy conversion +permits checking `PRAGMA freelist_count` and running VACUUM. A structure-only +upgrade does not repair notes in the worker or reclaim ordinary free pages. +Cancellation after conversion or a VACUUM failure can retry reclamation within +the same startup. After quitting, startup checks the schema and remaining legacy +payloads again; free pages alone do not trigger another upgrade, and SQLite can +reuse them. No additional persistent migration marker is stored. +General migration requirements are in the [database migration guide](../../development/database-migrations.md). ## Backup and Restore diff --git a/docs/quality/recurring-defects.md b/docs/quality/recurring-defects.md index bfb515e4..4bbc335c 100644 --- a/docs/quality/recurring-defects.md +++ b/docs/quality/recurring-defects.md @@ -7,6 +7,25 @@ 修正复发缺陷时，优先建立一条能覆盖整类问题的机制（共享选择器、共享函数、能发现全部实例 的测试），而不是逐个点名修补。逐个修补正是这些条目反复出现的原因。 +## 普通结构升级再次触发历史回收 + +**症状**：历史已经转换完成，后续升级仍显示清理历史并执行耗时的全库回收。 + +**原因**：只修复启动检测不够；普通旧 schema 仍会进入升级 Worker，若共享升级方法 +把任意空闲页当作回收依据，就会在以后每次结构升级时重现。 + +**规则**：共享启动决定分别描述内容转换与回收，普通结构升级默认不回收。只有具体 +旧内容转换允许回收，同次启动重试沿用内存决定；下一次启动重新检测，不增加持久标记。 +具体规则见[迁移指南](../development/database-migrations.md)。 + +**防复发**：schema 45／46 的真实 SQLite fixture 保留正常写入产生的空闲页，断言 +无历史转换、无 `VACUUM`、内容与完整性保留；真实 Worker 验证取消及重试， +启动调度测试分别覆盖允许和禁止回收的决定在重试中不变。 + +**历史**：`8dce1d7` 和 `38d0923` 先后修正启动检测，后者随 0.13.9 发布； +2026-09-26 检查 0.13.15 的 schema 46→47 路径发现共享 Worker 仍无条件按空闲页回收。 +本次修复在共享操作层分离结构升级与回收，而非给 46→47 增加特例。 + ## 异步界面测试只等待部分状态 **症状**：本地通过的界面测试在完整 CI 中找不到预期控件，或在前一步更新后立即读取到旧状态。 @@ -15,11 +34,15 @@ 按请求完成和最终可见状态同步，不以另一个元素出现或消失代替完整操作结束。 测试请求重试时显式控制后一次响应，分别断言等待中和响应完成后的状态； 同一异步操作的多个最终断言放入同一个 `waitFor`。不要只延长全局超时或放宽断言。 +Electron 原生拖拽测试应确认窗口焦点、拖拽状态建立、宽度更新及拖拽结束后再断言； +移动事件明确携带左键按下状态，不用固定睡眠代替输入处理完成。 **历史**：2026-09-26 检查 Desktop 0.13.14 标签 CI `36165805814` 与 main CI `36164698647`：原生能力重试用例在启动状态未稳定时打开选项， 浏览器关闭用例只等待页签消失便同步断言错误已清除。两处测试改用上述已有同步机制； 是否消除 CI 失败以修复提交的 Linux CI 结果为准。 +同日 Windows 全量测试及独立复跑中，Magic Notes 原生拖拽分别停在索引列 168 和 +AI 列 280。测试同步原生输入与实际状态后独立复跑通过，保留原有宽度和持久化断言。 ## 整窗遮罩与原生拖动区域 diff --git a/src/main/assistant-storage-startup.test.ts b/src/main/assistant-storage-startup.test.ts new file mode 100644 index 00000000..329bfae5 --- /dev/null +++ b/src/main/assistant-storage-startup.test.ts @@ -0,0 +1,77 @@ +import type { EventEmitter } from 'node:events' +import type { BrowserWindow, IpcMainInvokeEvent } from 'electron' +import { beforeEach, expect, it, vi } from 'vitest' +import { ipcChannels } from '../shared/ipc-channels' +import { prepareAssistantStorage } from './assistant-storage-startup' +import { getPendingAssistantStorageUpgrade } from './assistant/assistant-storage-upgrade' +import { loadMainWindow } from './window' + +const mocks = vi.hoisted(() => ({ + workers: [] as Array, + handlers: new Map unknown>(), + quit: vi.fn() +})) +vi.mock('electron', () => ({ + app: { quit: mocks.quit }, + ipcMain: { + handle: (channel: string, handler: (event: unknown, input?: unknown) => unknown) => mocks.handlers.set(channel, handler), + removeHandler: (channel: string) => mocks.handlers.delete(channel) + } +})) +vi.mock('node:fs/promises', () => { + const stat = vi.fn(async () => ({ size: 1024 })) + return { stat, default: { stat } } +}) +vi.mock('./window', () => ({ loadMainWindow: vi.fn() })) +vi.mock('./assistant/assistant-storage-upgrade', () => ({ getPendingAssistantStorageUpgrade: vi.fn() })) +vi.mock('node:worker_threads', async () => { + const { EventEmitter } = await import('node:events') + class Worker extends EventEmitter { + constructor(_path: string, readonly options: { workerData: { upgrade: unknown } }) { + super() + mocks.workers.push(this) + } + } + return { Worker, default: { Worker } } +}) +beforeEach(() => { + mocks.workers.length = 0 + mocks.handlers.clear() + vi.clearAllMocks() +}) + +it.each([false, true])('retains the original reclaimSpace=%s decision across startup retries', async (reclaimSpace) => { + const upgrade = { migrateNotes: reclaimSpace, reclaimSpace } + vi.mocked(getPendingAssistantStorageUpgrade).mockReturnValueOnce(upgrade).mockReturnValue(undefined) + const webContents = { mainFrame: { url: 'app://main' }, getURL: () => 'app://main' } + const window = { webContents, isDestroyed: () => false, setProgressBar: vi.fn() } as unknown as BrowserWindow + const event = { sender: webContents, senderFrame: webContents.mainFrame } as unknown as IpcMainInvokeEvent + const preparation = prepareAssistantStorage(window, '/test/assistant.sqlite', new AbortController().signal) + await vi.waitFor(() => expect(mocks.workers).toHaveLength(1)) + mocks.workers[0]!.emit('message', { + progress: { stage: 'upgrading', processed: 0, total: 0, bytesBefore: 1024 } + }) + expect(loadMainWindow).toHaveBeenCalledOnce() + expect(mocks.handlers.get(ipcChannels.storageUpgradeProgress)!(event)).toMatchObject({ stage: 'upgrading' }) + mocks.workers[0]!.emit('message', { error: 'Retry this startup' }) + mocks.workers[0]!.emit('exit', 0) + await vi.waitFor(() => + expect(mocks.handlers.get(ipcChannels.storageUpgradeProgress)!(event)).toMatchObject({ stage: 'failed' }) + ) + mocks.handlers.get(ipcChannels.storageUpgradeAction)!(event, 'retry') + await vi.waitFor(() => expect(mocks.workers).toHaveLength(2)) + expect(getPendingAssistantStorageUpgrade).toHaveBeenCalledOnce() + expect(mocks.workers.map(worker => worker.options.workerData.upgrade)).toEqual([upgrade, upgrade]) + mocks.workers[1]!.emit('message', { done: true }) + mocks.workers[1]!.emit('exit', 0) + await preparation + expect(mocks.handlers.size).toBe(0) + expect(window.setProgressBar).toHaveBeenLastCalledWith(-1) +}) + +it('does not open an upgrade page or worker for an already-current database', async () => { + vi.mocked(getPendingAssistantStorageUpgrade).mockReturnValue(undefined) + await prepareAssistantStorage({} as BrowserWindow, '/test/assistant.sqlite', new AbortController().signal) + expect(mocks.workers).toHaveLength(0) + expect(loadMainWindow).not.toHaveBeenCalled() +}) diff --git a/src/main/assistant-storage-startup.ts b/src/main/assistant-storage-startup.ts index 87b61ec4..cc71abd2 100644 --- a/src/main/assistant-storage-startup.ts +++ b/src/main/assistant-storage-startup.ts @@ -10,7 +10,7 @@ import { } from '../shared/assistant-storage-contracts' import { ipcChannels } from '../shared/ipc-channels' import { assertTrustedSender } from './trusted-ipc-sender' -import { hasPendingAssistantStorageUpgrade } from './assistant/assistant-storage-upgrade' +import { getPendingAssistantStorageUpgrade, type AssistantStorageUpgrade } from './assistant/assistant-storage-upgrade' import { loadMainWindow } from './window' export async function prepareAssistantStorage( @@ -23,9 +23,10 @@ export async function prepareAssistantStorage( throw error }) if (!file) return - if (!hasPendingAssistantStorageUpgrade(databasePath)) return + const upgrade = getPendingAssistantStorageUpgrade(databasePath) + if (!upgrade) return let progress: AssistantStorageProgress = { - stage: 'scanning', processed: 0, total: 0, bytesBefore: file.size + stage: 'upgrading', processed: 0, total: 0, bytesBefore: file.size } let shown = false let retry: (() => void) | undefined @@ -53,13 +54,13 @@ export async function prepareAssistantStorage( try { while (!signal.aborted) { try { - await runStorageWorker(databasePath, signal, publish) + await runStorageWorker(databasePath, upgrade, signal, publish) return } catch (error) { if (signal.aborted) return publish({ ...progress, stage: 'failed', - error: error instanceof Error ? error.message : '历史执行记录优化失败' + error: error instanceof Error ? error.message : '本地数据升级失败' }) await new Promise((resolve) => { const finish = (): void => { @@ -81,6 +82,7 @@ export async function prepareAssistantStorage( function runStorageWorker( databasePath: string, + upgrade: AssistantStorageUpgrade, signal: AbortSignal, publish: (progress: AssistantStorageProgress) => void ): Promise { @@ -93,7 +95,7 @@ function runStorageWorker( return new Promise((resolve, reject) => { const worker = new Worker(join( dirname(fileURLToPath(import.meta.url)), 'assistant-storage-worker.js' - ), { workerData: { databasePath, cancellation } }) + ), { workerData: { databasePath, cancellation, upgrade } }) let failure: Error | undefined let done = false worker.on('message', (message: { @@ -107,13 +109,13 @@ function runStorageWorker( if (message.done || message.cancelled) done = true }) worker.once('error', () => { - failure = new Error('无法启动历史执行记录优化，请重启 GoodBuddy 后重试。') + failure = new Error('无法启动本地数据升级，请重启 GoodBuddy 后重试。') }) worker.once('exit', (code) => { signal.removeEventListener('abort', cancel) if (failure) reject(failure) else if (done && code === 0) resolve() - else reject(new Error('历史执行记录优化意外结束，请重试。')) + else reject(new Error('本地数据升级意外结束，请重试。')) }) }) } diff --git a/src/main/assistant-storage-worker.ts b/src/main/assistant-storage-worker.ts index c178c830..c3a1f0cb 100644 --- a/src/main/assistant-storage-worker.ts +++ b/src/main/assistant-storage-worker.ts @@ -1,25 +1,26 @@ import { parentPort, workerData } from 'node:worker_threads' -import { upgradeAssistantStorage } from './assistant/assistant-storage-upgrade' +import { upgradeAssistantStorage, type AssistantStorageUpgrade } from './assistant/assistant-storage-upgrade' const input = workerData as { databasePath: string cancellation: SharedArrayBuffer + upgrade: AssistantStorageUpgrade } const cancellation = new Int32Array(input.cancellation) try { - // Startup checks for legacy data before spawning this worker, including retries. + // Preserve this startup's actual conversion/reclamation decision across retries. upgradeAssistantStorage( input.databasePath, (progress) => parentPort?.postMessage({ progress }), () => Atomics.load(cancellation, 0) !== 0, - { pendingUpgradeConfirmed: true } + { confirmedUpgrade: input.upgrade } ) parentPort?.postMessage({ done: true }) } catch (error) { parentPort?.postMessage({ error: error instanceof Error && /full|space/iu.test(error.message) ? '磁盘空间不足，无法完成空间回收。请释放一些空间后重试。' - : '历史执行记录优化未完成。请重试；若仍失败，请备份数据并联系支持。', + : '本地数据升级未完成。请重试；若仍失败，请备份数据并联系支持。', cancelled: error instanceof Error && error.name === 'AbortError' }) } diff --git a/src/main/assistant/activity-history-io.test.ts b/src/main/assistant/activity-history-io.test.ts index 7bcf9616..4895384e 100644 --- a/src/main/assistant/activity-history-io.test.ts +++ b/src/main/assistant/activity-history-io.test.ts @@ -6,7 +6,7 @@ import { join } from 'node:path' import { DatabaseSync } from 'node:sqlite' import { expect, it, vi } from 'vitest' import { AssistantDatabase } from './assistant-database' -import { upgradeAssistantStorage } from './assistant-storage-upgrade' +import { getPendingAssistantStorageUpgrade, upgradeAssistantStorage } from './assistant-storage-upgrade' it('migrates activity history and writes only changed records during streaming', () => { const directory = mkdtempSync(join(tmpdir(), 'goodbuddy-save-io-')) @@ -41,6 +41,7 @@ it('migrates activity history and writes only changed records during streaming', } const beforeBytes = statSync(`${path}-wal`).size database.close() + expect(getPendingAssistantStorageUpgrade(path)).toEqual({ migrateNotes: false, reclaimSpace: true }) upgradeAssistantStorage(path, () => undefined) database.initialize(directory) expect(database.getActivityHistory()).toEqual({ records, legacyHistoryMayBeIncomplete: true }) diff --git a/src/main/assistant/assistant-database.ts b/src/main/assistant/assistant-database.ts index 6dcdabd4..5333f45d 100644 --- a/src/main/assistant/assistant-database.ts +++ b/src/main/assistant/assistant-database.ts @@ -1846,21 +1846,24 @@ export class AssistantDatabase { } // Called by the existing startup worker before the business connection opens. - upgradeMagicNoteStorage( + upgradeStorage( onProgress: (progress: AssistantStorageProgress) => void, - isCancelled: () => boolean + isCancelled: () => boolean, + upgrade: { migrateNotes: boolean; reclaimSpace: boolean } ): void { const database = new DatabaseSync(this.databasePath, { timeout: 5_000 }) try { database.exec('PRAGMA journal_mode = WAL; PRAGMA foreign_keys = ON;') this.migrate(database, onProgress, isCancelled) this.database = database - this.repairMagicNoteStorage(onProgress, isCancelled) + if (upgrade.migrateNotes) this.repairMagicNoteStorage(onProgress, isCancelled) if (isCancelled()) throw new DOMException('Upgrade cancelled', 'AbortError') - const free = database.prepare('PRAGMA freelist_count').get() as { freelist_count: number } - if (free.freelist_count > 0) { - onProgress({ stage: 'compacting', processed: 0, total: 0, bytesBefore: statSync(this.databasePath).size }) - database.exec('PRAGMA wal_checkpoint(TRUNCATE); VACUUM;') + if (upgrade.reclaimSpace) { + const free = database.prepare('PRAGMA freelist_count').get() as { freelist_count: number } + if (free.freelist_count > 0) { + onProgress({ stage: 'compacting', processed: 0, total: 0, bytesBefore: statSync(this.databasePath).size }) + database.exec('PRAGMA wal_checkpoint(TRUNCATE); VACUUM;') + } } database.exec('PRAGMA wal_checkpoint(TRUNCATE)') } finally { diff --git a/src/main/assistant/assistant-storage-upgrade.test.ts b/src/main/assistant/assistant-storage-upgrade.test.ts index d83b0a15..a59e463e 100644 --- a/src/main/assistant/assistant-storage-upgrade.test.ts +++ b/src/main/assistant/assistant-storage-upgrade.test.ts @@ -9,6 +9,7 @@ import type { ConversationMessageBlock } from '../../shared/assistant-contracts' import type { SubagentEvent } from '../../shared/contracts' import { AssistantDatabase, ASSISTANT_DATABASE_SCHEMA_VERSION } from './assistant-database' import { + getPendingAssistantStorageUpgrade, hasPendingAssistantStorageUpgrade, upgradeAssistantStorage } from './assistant-storage-upgrade' @@ -54,6 +55,58 @@ function subagent(requestId: string): SubagentEvent { } describe('subagent progress storage', () => { + it.each([45, 46])('upgrades schema %i without reconverting history or vacuuming ordinary free pages', async (sourceVersion) => { + const { path, database, taskId } = await fixture() + const event = subagent(randomUUID()) + event.progress = [{ id: randomUUID(), type: 'text', content: 'Keep current-format progress' }] + database.appendTaskEvent(taskId, 'subagent', event) + const note = database.createMagicNote({ title: 'Already migrated', content: { version: 1, ops: [{ insert: 'Keep note\n' }] } }) + const header = { id: randomUUID(), title: 'Keep chat', updatedAt: 1000 } + const message = { id: randomUUID(), role: 'assistant' as const, state: 'complete' as const, + content: 'Normal chat update'.repeat(100_000), createdAt: 1000 } + database.saveLocalConversations([{ header, messages: [message] }]) + database.saveLocalConversations([{ header, messages: [{ ...message, content: 'Final response' }] }]) + const expected = database.getConversation(header.id) + database.close() + const sql = new DatabaseSync(path) + try { + sql.exec(`DROP VIEW supervision_review_current; + DROP TABLE supervision_review_navigation; DROP TABLE supervision_review_batches; + DROP TABLE supervision_review_sources; DROP TABLE supervision_review_runs;`) + if (sourceVersion === 45) sql.exec(` + DROP TRIGGER messages_review_insert; DROP TRIGGER messages_review_update; + DROP TRIGGER messages_review_delete; DROP TRIGGER tasks_review_delete; + DROP TABLE review_checkpoints; ALTER TABLE messages DROP COLUMN review_revision;`) + sql.exec(`PRAGMA user_version = ${sourceVersion}; PRAGMA wal_checkpoint(TRUNCATE)`) + expect(sql.prepare('PRAGMA freelist_count').get()!.freelist_count).toBeGreaterThan(0) + const events = sql.prepare('SELECT * FROM task_events WHERE task_id = ? ORDER BY id').all(taskId) + expect(events.length).toBeGreaterThan(0) + expect(hasPendingAssistantStorageUpgrade(path)).toBe(true) + const confirmedUpgrade = getPendingAssistantStorageUpgrade(path)! + expect(confirmedUpgrade).toEqual({ migrateNotes: false, reclaimSpace: false }) + const exec = vi.spyOn(DatabaseSync.prototype, 'exec') + const progress = vi.fn() + upgradeAssistantStorage(path, progress) + expect(exec.mock.calls.some(([statement]) => /VACUUM/i.test(statement))).toBe(false) + expect(progress.mock.calls.map(([value]) => value.stage)).not.toContain('compacting') + expect(progress.mock.calls.map(([value]) => value.stage)).not.toContain('converting') + expect(sql.prepare('PRAGMA user_version').get()!.user_version).toBe(ASSISTANT_DATABASE_SCHEMA_VERSION) + expect(sql.prepare('SELECT * FROM task_events WHERE task_id = ? ORDER BY id').all(taskId)).toEqual(events) + // A retry in the same startup must not gain reclamation just because free pages remain. + upgradeAssistantStorage(path, progress, () => false, { confirmedUpgrade }) + expect(exec.mock.calls.some(([statement]) => /VACUUM/i.test(statement))).toBe(false) + database.initialize(dirname(path)) + expect(database.getConversation(header.id)).toEqual(expected) + expect(database.getMagicNote(note.id)).toEqual(note) + database.close() + progress.mockClear() + upgradeAssistantStorage(path, progress) + expect(progress).not.toHaveBeenCalled() + expect(hasPendingAssistantStorageUpgrade(path)).toBe(false) + expect(sql.prepare('PRAGMA integrity_check').get()).toEqual({ integrity_check: 'ok' }) + } finally { sql.close() } + }) + it('does not migrate or vacuum ordinary chat free pages on repeated startup', async () => { const { path, database } = await fixture() const header = { id: randomUUID(), title: 'Chat restart', updatedAt: 1000 } diff --git a/src/main/assistant/assistant-storage-upgrade.ts b/src/main/assistant/assistant-storage-upgrade.ts index b9a1083c..824527d2 100644 --- a/src/main/assistant/assistant-storage-upgrade.ts +++ b/src/main/assistant/assistant-storage-upgrade.ts @@ -12,9 +12,14 @@ import { SUBAGENT_PROGRESS_STORAGE_SCHEMA_VERSION } from './subagent-progress-storage' -export function hasPendingAssistantStorageUpgrade( +export type AssistantStorageUpgrade = { + migrateNotes: boolean + reclaimSpace: boolean +} + +export function getPendingAssistantStorageUpgrade( databasePath: string -): boolean { +): AssistantStorageUpgrade | undefined { const database = new DatabaseSync(databasePath, { readOnly: true, timeout: 5_000 @@ -23,37 +28,50 @@ export function hasPendingAssistantStorageUpgrade( const row = database.prepare('PRAGMA user_version').get() as { user_version: number } - if (row.user_version < ASSISTANT_DATABASE_SCHEMA_VERSION) { - return true - } const noteTable = database.prepare( `SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'magic_note_entries' LIMIT 1` ).get() - if (!noteTable) return false - return Boolean(database.prepare( + const migrateNotes = Boolean(noteTable && database.prepare( `SELECT 1 FROM magic_note_entries WHERE json_extract(content_json, '$.storage') IS NULL OR json_extract(content_json, '$.storage') <> 'file' LIMIT 1` ).get()) + if (row.user_version >= ASSISTANT_DATABASE_SCHEMA_VERSION && !migrateNotes) return undefined + // Reclamation belongs to specific released data conversions, not every schema bump. + const legacySubagents = row.user_version > 0 && + row.user_version < SUBAGENT_PROGRESS_STORAGE_SCHEMA_VERSION && + Boolean(database.prepare("SELECT 1 FROM task_events WHERE kind = 'subagent' LIMIT 1").get()) + const legacyActivity = row.user_version > 0 && row.user_version < 44 && + Boolean(database.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'activity_history'").get()) && + Boolean(database.prepare("SELECT 1 FROM activity_history WHERE records_json <> '[]' LIMIT 1").get()) + return { migrateNotes, reclaimSpace: migrateNotes || legacySubagents || legacyActivity } } finally { database.close() } } +export function hasPendingAssistantStorageUpgrade(databasePath: string): boolean { + return getPendingAssistantStorageUpgrade(databasePath) !== undefined +} + export function upgradeAssistantStorage( databasePath: string, onProgress: (progress: AssistantStorageProgress) => void, isCancelled: () => boolean = () => false, - options: { pendingUpgradeConfirmed?: boolean } = {} + options: { confirmedUpgrade?: AssistantStorageUpgrade } = {} ): void { - if (!options.pendingUpgradeConfirmed && !hasPendingAssistantStorageUpgrade(databasePath)) return + const upgrade = options.confirmedUpgrade ?? getPendingAssistantStorageUpgrade(databasePath) + if (!upgrade) return + if (isCancelled()) throw new DOMException('Upgrade cancelled', 'AbortError') upgradeSubagentStorage(databasePath, onProgress, isCancelled) - new AssistantDatabase(databasePath).upgradeMagicNoteStorage(onProgress, isCancelled) + onProgress({ stage: 'upgrading', processed: 0, total: 0, bytesBefore: statSync(databasePath).size }) + if (isCancelled()) throw new DOMException('Upgrade cancelled', 'AbortError') + new AssistantDatabase(databasePath).upgradeStorage(onProgress, isCancelled, upgrade) } function upgradeSubagentStorage( @@ -137,7 +155,7 @@ function upgradeSubagentStorage( const free = database.prepare('PRAGMA freelist_count').get() as { freelist_count: number } - if (free.freelist_count > 0) { + if (tasks.length > 0 && free.freelist_count > 0) { progress.stage = 'compacting' onProgress({ ...progress }) // Rebuild the now-small live database, not a second copy of the bloated input. diff --git a/src/main/magic-notes/magic-note-storage.test.ts b/src/main/magic-notes/magic-note-storage.test.ts index 0e53ba06..d3e2310d 100644 --- a/src/main/magic-notes/magic-note-storage.test.ts +++ b/src/main/magic-notes/magic-note-storage.test.ts @@ -6,7 +6,7 @@ import { backup, DatabaseSync } from 'node:sqlite' import { afterEach, describe, expect, it, vi } from 'vitest' import type { MagicNoteCanvasContent, MagicNoteContent } from '../../shared/magic-notes-contracts' import { AssistantDatabase, ASSISTANT_DATABASE_SCHEMA_VERSION } from '../assistant/assistant-database' -import { upgradeAssistantStorage } from '../assistant/assistant-storage-upgrade' +import { getPendingAssistantStorageUpgrade, upgradeAssistantStorage } from '../assistant/assistant-storage-upgrade' const actualFs = { renameSync: fs.renameSync, rmSync: fs.rmSync } @@ -150,6 +150,7 @@ describe('Magic note SQLite and filesystem storage', () => { database.close() sql.prepare('UPDATE magic_note_entries SET content_json = ? WHERE id = ?').run(JSON.stringify(note.entries[0]!.content), note.entries[0]!.id) sql.exec('PRAGMA wal_checkpoint(TRUNCATE)') + const confirmedUpgrade = getPendingAssistantStorageUpgrade(path)! let cancelled = false const exec = DatabaseSync.prototype.exec const injected = vi.spyOn(DatabaseSync.prototype, 'exec').mockImplementation(function (this: DatabaseSync, statement) { @@ -162,9 +163,10 @@ describe('Magic note SQLite and filesystem storage', () => { injected.mockRestore() expect(sql.prepare("SELECT json_extract(content_json, '$.storage') AS storage FROM magic_note_entries").get()!.storage).toBe('file') expect(sql.prepare('PRAGMA freelist_count').get()!.freelist_count).toBeGreaterThan(0) + expect(getPendingAssistantStorageUpgrade(path)).toBeUndefined() const stages: string[] = [] upgradeAssistantStorage(path, (progress) => stages.push(progress.stage), () => false, { - pendingUpgradeConfirmed: true + confirmedUpgrade }) expect(stages).toContain('compacting') expect(sql.prepare('PRAGMA freelist_count').get()!.freelist_count).toBe(0) @@ -257,7 +259,9 @@ describe('Magic note SQLite and filesystem storage', () => { expect(JSON.parse(String(sql.prepare('SELECT content_json FROM magic_note_entries WHERE id = ?').get(entries[0]!.id)!.content_json))).toEqual(rich) sql.exec('DROP TRIGGER fail_note_migration') let cancelled = false - expect(() => upgradeAssistantStorage(path, () => { cancelled = true }, () => cancelled)).toThrow('cancelled') + expect(() => upgradeAssistantStorage(path, (progress) => { + if (progress.stage === 'converting') cancelled = true + }, () => cancelled)).toThrow('cancelled') expect(sql.prepare("SELECT COUNT(*) AS count FROM magic_note_entries WHERE json_extract(content_json, '$.storage') = 'file'").get()!.count).toBe(1) upgradeAssistantStorage(path, () => undefined) database.initialize(directory) diff --git a/src/renderer/src/StorageUpgrade.test.tsx b/src/renderer/src/StorageUpgrade.test.tsx index 6a47dc0e..5274fbf6 100644 --- a/src/renderer/src/StorageUpgrade.test.tsx +++ b/src/renderer/src/StorageUpgrade.test.tsx @@ -2,9 +2,40 @@ import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/re import { afterEach, expect, it, vi } from 'vitest' import { StorageUpgrade } from './StorageUpgrade' import type { DesktopApi } from '../../shared/contracts' +import { UiLocaleProvider } from './i18n/UiLocaleProvider' afterEach(() => { cleanup(); vi.unstubAllGlobals() }) +it.each(['zh-CN', 'en-US'] as const)('describes schema-only upgrades without claiming history conversion or reclamation (%s)', async (locale) => { + vi.stubGlobal('goodbuddy', { storageUpgrade: { + getProgress: vi.fn(async () => ({ + stage: 'upgrading', processed: 0, total: 0, bytesBefore: 100_000 + })), + act: vi.fn(async () => undefined) + } } as unknown as DesktopApi) + render() + expect(await screen.findByRole('status')).toHaveTextContent(locale === 'zh-CN' + ? '正在更新数据库结构' : 'Applying database structure updates') + expect(screen.getByRole('heading')).toHaveTextContent(locale === 'zh-CN' + ? '正在升级本地数据结构' : 'Upgrading local data structures') + expect(screen.queryByText(/清理子任务进度的重复副本|removes repeated copies of subagent progress/)).not.toBeInTheDocument() + expect(screen.queryByRole('progressbar')).not.toBeInTheDocument() +}) + +it.each(['zh-CN', 'en-US'] as const)('only describes space reclamation when it is actually running (%s)', async (locale) => { + vi.stubGlobal('goodbuddy', { storageUpgrade: { + getProgress: vi.fn(async () => ({ + stage: 'compacting', processed: 0, total: 0, bytesBefore: 100_000 + })), + act: vi.fn(async () => undefined) + } } as unknown as DesktopApi) + render() + await waitFor(() => expect(screen.getByRole('status')).toHaveTextContent(locale === 'zh-CN' + ? '正在回收磁盘空间' : 'Reclaiming disk space')) + expect(screen.getByRole('heading')).toHaveTextContent(locale === 'zh-CN' + ? '正在回收数据库空间' : 'Reclaiming database space') +}) + it('shows real progress and lets the user quit without deleting records', async () => { const act = vi.fn(async () => undefined) vi.stubGlobal('goodbuddy', { storageUpgrade: { diff --git a/src/renderer/src/StorageUpgrade.tsx b/src/renderer/src/StorageUpgrade.tsx index 7be555cf..de8e76a4 100644 --- a/src/renderer/src/StorageUpgrade.tsx +++ b/src/renderer/src/StorageUpgrade.tsx @@ -40,17 +40,24 @@ export function StorageUpgrade(): React.JSX.Element { void poll() return () => { active = false; clearTimeout(timer) } }, []) - const title = en ? 'Optimizing saved execution history' : '正在优化历史执行记录' const stage = progress?.stage + const convertingHistory = stage === 'scanning' || stage === 'converting' + const title = stage === 'compacting' + ? en ? 'Reclaiming database space' : '正在回收数据库空间' + : convertingHistory + ? en ? 'Upgrading saved history' : '正在转换旧版历史数据' + : en ? 'Upgrading local data structures' : '正在升级本地数据结构' const status = stage === 'failed' ? progress?.error : stage === 'compacting' ? en ? 'Reclaiming disk space…' : '正在回收磁盘空间…' : stage === 'complete' - ? en ? 'Optimization complete. Starting GoodBuddy…' : '优化完成，正在启动 GoodBuddy…' + ? en ? 'Data upgrade complete. Starting GoodBuddy…' : '数据升级完成，正在启动 GoodBuddy…' : progress?.stage === 'converting' ? `${en ? 'Processed' : '已处理'} ${progress.processed.toLocaleString()} / ${progress.total.toLocaleString()}` - : en ? 'Checking saved execution history…' : '正在检查历史执行记录…' + : stage === 'scanning' + ? en ? 'Checking saved history…' : '正在检查旧版历史数据…' + : en ? 'Applying database structure updates…' : '正在更新数据库结构…' const act = async (action: 'retry' | 'quit'): Promise => { try { setActionError('') @@ -65,9 +72,17 @@ export function StorageUpgrade(): React.JSX.Element {

{status}

{progress?.stage === 'converting' && progress.total > 0 && ( @@ -77,8 +92,8 @@ export function StorageUpgrade(): React.JSX.Element {

{en ? 'Processing speed' : '处理速度'}：{Math.round(rate).toLocaleString()} {en ? 'events/s' : '条/秒'}

)}

{en - ? 'You can quit and retry on the next launch. Completed batches are preserved.' - : '可以退出并在下次启动时继续处理，已完成的批次会保留。'}

+ ? 'You can quit and continue any remaining required upgrades on the next launch. Committed changes are preserved.' + : '可以退出，下次启动时继续尚未完成的必要升级；已提交的变更会保留。'}

{actionError &&

{actionError}

} {stage === 'failed' && (