diff --git a/docs/features/document-processing/README.md b/docs/features/document-processing/README.md index a644d978..42034843 100644 --- a/docs/features/document-processing/README.md +++ b/docs/features/document-processing/README.md @@ -7,7 +7,7 @@ | 文档 | 权威职责 | | --- | --- | | [文档解析与本地 OCR PRD](./prd.md) | 产品范围、解析策略、OCR 模型管理和验收 | -| [聊天附件技术说明](./chat-attachments-technical-design.md) | 文件选择与粘贴的进程链路、IPC、失败处理和验证方法 | +| [聊天附件技术说明](./chat-attachments-technical-design.md) | 文件选择与粘贴的进程链路、IPC、PPTX 图片 OCR、Worker 生命周期和验证证据 | 当前尚无独立 User Stories、功能逻辑、UI 和进度文档；聊天输入区交互遵循根目录 [UI 设计系统](../../../UI-DESIGN.md)。 diff --git a/docs/features/document-processing/chat-attachments-technical-design.md b/docs/features/document-processing/chat-attachments-technical-design.md index a2df6553..689e1a65 100644 --- a/docs/features/document-processing/chat-attachments-technical-design.md +++ b/docs/features/document-processing/chat-attachments-technical-design.md @@ -24,6 +24,27 @@ Main。IPC 先验证可信 sender，再使用共享 `contextImportFilesSchema` 会话草稿；发送时通过已有 `contextIds` 和 `ContextManager.enrichRequest` 生成请求上下文。 本次变更不修改 Agent、远端协议或 Runtime 的附件消费实现。 +## PPTX 图片识别与 OCR 生命周期 + +产品策略与限制分别见 [PPTX 图片流程](./prd.md#75-pptx-图片流程)、 +[OCR 空闲资源回收](./prd.md#76-ocr-空闲资源回收)。 + +`DocumentParsingService` 在非快速工作流中使用 `knowledge/pptx-parser.ts` 读取 +演示文稿顺序、幻灯片文字、图片关系和内嵌图片。Office ZIP 读取复用 +`document-parser.ts` 的条目数量、单条目及总展开大小检查；不读取工作区中的关联文件。 +每张引用图片通过现有 `DocumentOcrBroker` 的图片请求契约，进入生产 +`document-ocr-bridge.ts` 和 WASM Worker。结果合并时保留原生文字，附上幻灯片页码、 +图片定位及 OCR 置信度，诊断中的 OCR 页数按页码去重。 + +Bridge 用待完成请求计数判断队列是否空闲，正常完成后安排一次空闲计时； +新请求和 `terminateWorker` 均清除计时。到期通过 `Worker.terminate()` 释放整个推理环境， +而不是只清空 JavaScript 引用或反复触发垃圾回收。此机制由 PDF 与 PPTX 共用， +不增加 IPC、持久化设置或模型安装状态。 + +解析与回收发生在 Desktop 的 Main/Renderer，`gbagent` 不运行此 OCR Worker。 +远程 Runtime 消费的仍是 Desktop 已提取的附件文本，没有单独的 Agent 解析器或 +需要同步修改的远程 OCR 生命周期。 + ## Electron 验证 2026-09-15 在 Windows 的 Electron 43.2.0 上，从当前源码构建隔离输出并启动完整应用， @@ -67,3 +88,28 @@ npm run lint 沙箱用例，类型检查与 lint 通过。完整 `npm test` 运行结果为 4214 项通过、67 项跳过、 6 项窗口恢复用例失败；这些用例属于并行修改的 `window.test.ts`，随后该文件单独复跑 10 项全部通过。本次没有再次完整重跑全仓测试。 + +## 2026-09-15 PPTX OCR 与空闲回收验证 + +在 Windows x64、Electron 43.2.0 上，从本次源码构建独立输出，使用独立用户数据目录和 +本机已安装的 PP-OCRv6 Tiny 模型完成真实验证。测试仅替换系统文件选择对话框的返回路径， +保留生产 UI、preload、IPC、`ContextManager`、解析服务、OCR Broker 与 WASM Worker， +未替换解析或推理实现。 + +- 从“添加附件”一次导入 4 份图片型 PPTX，共 40 页；4 个附件均成功显示，连续处理期间 + 仅创建 1 个 OCR Worker。 +- 队列空闲后观察到 Worker 被终止。每 5 秒读取 Electron `app.getAppMetrics()`： + Renderer 工作集采样高点为 733,492 KiB（约 716 MiB），回收后为 162,716 KiB + （约 159 MiB），启动基线约 163 MiB。该结果是本次设备与文件上的测量，不是固定内存承诺。 +- 回收后，通过生产 `context.importFiles` 再导入一份 9 页 PPTX；新建第 2 个 Worker， + 9 页均返回非空文字，共 2,462 字符，附件成功生成。 +- 本次合计 49 次本地图片 OCR 请求，文本模型调用 0 次；没有发送聊天消息，也没有将文件 + 交给云端模型。未修改源文件。未在 macOS、Linux 或其他 OCR 模型档位上做本次实测。 + +新增聚焦回归覆盖 PPTX 关系与顺序、各场景路由、原生文字合并、快速模式、页数上限、 +OCR 错误和取消；Worker 回归覆盖空闲到期、连续请求复用、队列保护、关闭清理与立即取消。 +空闲回收后重新加载也在真实模型路径上验证，不能只用模型配置检查代替。 + +最终检查：`npm test` 4,293 项通过、67 项跳过；`npm run typecheck`、 +`npm run lint` 通过。当前源码的隔离 Electron 生产构建通过；文档的 10 个相对文件链接 +均有效。测试应用已关闭，包含附件草稿的独立用户数据目录已删除，原文件和原模型安装未改动。 diff --git a/docs/features/document-processing/prd.md b/docs/features/document-processing/prd.md index a8153723..7648cdda 100644 --- a/docs/features/document-processing/prd.md +++ b/docs/features/document-processing/prd.md @@ -22,13 +22,16 @@ GoodBuddy 需要用同一条可信文档解析链路服务以下场景： - XLSX 工作表 XML 与共享字符串； - PPTX 幻灯片文字。 +聊天附件、知识库和测试解析通过统一的 `DocumentParsingService` 使用已保存的工作流。 +扫描 PDF 和 PPTX 幻灯片内嵌图片可使用已安装、已校验的本地 OCR 模型识别； +PPTX 的当前能力和边界见 [PPTX 图片流程](#75-pptx-图片流程)。 + 现有局限： -- 纯扫描 PDF 没有文本层时无法提取内容； - DOC、XLS、PPT 等旧版二进制 Office 格式不支持； - Office 解析主要提取文字，不能完整保留表格、公式、图表和版面； -- 聊天附件和知识库直接调用底层解析函数，缺少可配置的统一工作流； -- 没有本地 OCR 模型状态、真实解析测试和按场景策略。 +- DOCX、XLSX 内嵌图片尚未接入 OCR； +- Office 格式转换和远程 OCR 服务尚未接入。 ## 3. 产品原则 @@ -277,6 +280,32 @@ ZIP 根目录包含模型文件和 `goodbuddy-model.json`。清单格式为 `goo 受密码保护、损坏或超限的 PDF 不得进入 OCR。 +### 7.5 PPTX 图片流程 + +自动解析、完整索引和高保真工作流保留幻灯片原生文字，同时对幻灯片直接引用的 +内嵌 PNG、JPEG、WebP 图片运行本地 OCR。即使幻灯片已有标题或正文，也不会因此 +跳过图片中的文字。快速文本、快速索引工作流不运行 OCR；图片型 PPTX 在这些模式下 +明确提示当前工作流未启用 OCR。 + +按演示文稿中的幻灯片顺序返回结果，图片识别结果标注 `幻灯片 N · 图片 M`， +保留页码和识别置信度。OCR 页数限制计算含图片的幻灯片数，而不是图片总数； +图片顺序处理并复用已加载模型，沿用现有超时、取消和输出限制。 + +这不是 Office 渲染或转换：不识别母版、版式中继承的图片，不重建裁剪、旋转或遮挡后的 +完整画面，不把演讲者备注当作幻灯片正文，也不下载外链图片。其他内嵌图片格式在 +需要 OCR 时明确报错。自动聊天解析可在 OCR 失败时保留可靠原生文字并附带警告； +图片型文件没有可靠原生文字时仍失败，知识库和高保真工作流不以此回退掩盖 OCR 失败。 + +### 7.6 OCR 空闲资源回收 + +OCR 队列清空且没有正在处理的请求后，保留 Worker 60 秒，以复用连续文档的模型。 +期间收到新请求立即取消回收计时；请求完成、队列再次空闲后重新计时。空闲到期后 +销毁 Worker，释放其模型会话和 WASM 运行环境；下一次请求按需重新加载已安装模型。 + +回收不会删除模型文件或已经解析的附件文字，不打断正在识别、初始化或排队的任务。 +取消、超时和服务关闭仍按原有路径立即清理。操作系统显示的进程内存不保证立刻恢复到 +启动值，不能仅凭一次内存快照判断是否发生泄漏。 + ## 8. Office 与转换 本节只定义 Office 文件的语义解析和格式转换，不提供可视化编辑器、编辑会话、选区、撤销或 @@ -361,6 +390,7 @@ DOC、XLS、PPT 通过 `DocumentConversionProvider` 转换： ## 12. 验收 - 同一份扫描 PDF 可从聊天附件和知识库得到一致的逐页文本； +- 图片型 PPTX 在自动解析下使用已配置的本地 OCR，并保留幻灯片来源定位； - 文本型 PDF 在 `auto` 模式下不运行 OCR； - 本地 OCR 在六个平台和两种架构上完全离线运行； - 模型文件损坏时拒绝加载并显示可恢复错误； @@ -371,6 +401,7 @@ DOC、XLS、PPT 通过 `DocumentConversionProvider` 转换： - 语音和 OCR 模型可在联网设备导出 ZIP，并在离线设备导入后完成真实推理； - 路径穿越、未知条目、错误模型 ID、篡改文件和超限 ZIP 均被拒绝； - 超页数、超时、取消和关闭不会留下运行任务； +- OCR 空闲 60 秒后释放 Worker，连续请求复用模型，回收后的下一次解析可重新加载模型完成； - 测试解析不会创建聊天消息或知识库文档； - 选择本地模型时没有任何文档上传； - 文档中的提示词不会改变系统、模式或工具权限。 diff --git a/src/main/document-parsing-service.test.ts b/src/main/document-parsing-service.test.ts index b2301001..3fa571d9 100644 --- a/src/main/document-parsing-service.test.ts +++ b/src/main/document-parsing-service.test.ts @@ -3,6 +3,7 @@ import { defaultDocumentParsingSettings } from './document-parsing-settings-store' import { DocumentParsingService } from './document-parsing-service' +import { createImagePptx } from '../../tests/support/pptx-fixture' function createPdfFixture(...pageTexts: string[]): Buffer { const texts = pageTexts.length > 0 ? pageTexts : [''] @@ -109,6 +110,73 @@ function createService(overrides?: { } describe('DocumentParsingService', () => { + it.each(['chat-attachment', 'knowledge-index', 'artifact-import', 'diagnostic'] as const)( + 'recognizes embedded PPTX images for %s', + async (purpose) => { + const { recognize, service } = createService() + const parsed = await service.parse('slides.pptx', createImagePptx('', 2), purpose) + expect(parsed.sourceFormat).toBe('.pptx') + expect(parsed.pageCount).toBe(2) + expect(parsed.sections.map(({ locator, method, pageNumber }) => ({ + locator, method, pageNumber + }))).toEqual([ + { locator: '幻灯片 1 · 图片 1', method: 'ocr', pageNumber: 1 }, + { locator: '幻灯片 2 · 图片 1', method: 'ocr', pageNumber: 2 } + ]) + expect(recognize).toHaveBeenCalledTimes(2) + expect(recognize).toHaveBeenCalledWith(expect.objectContaining({ + mimeType: 'image/png', maximumPages: 1, modelId: 'pp-ocrv6-tiny' + })) + } + ) + + it('preserves PPTX native text while recognizing image content on the same slide', async () => { + const { service } = createService() + const parsed = await service.parse('slides.pptx', createImagePptx('Native slide title'), 'chat-attachment') + expect(parsed.content).toContain('Native slide title') + expect(parsed.content).toContain('扫描件识别正文') + const diagnostic = await service.diagnose('slides.pptx', createImagePptx('Native slide title')) + expect(diagnostic).toMatchObject({ method: 'mixed', pageCount: 1, ocrPageCount: 1 }) + }) + + it('reports disabled OCR for image-only PPTX in fast-text mode', async () => { + const { service, recognize } = createService({ settings: { chatWorkflow: 'fast-text' } }) + await expect(service.parse('slides.pptx', createImagePptx(), 'chat-attachment')).rejects.toThrow('当前工作流未启用 OCR') + expect(recognize).not.toHaveBeenCalled() + }) + + it('enforces PPTX OCR page limits before recognizing images', async () => { + const { service, recognize } = createService({ settings: { maximumPages: 1 } }) + await expect(service.parse('slides.pptx', createImagePptx('', 2), 'knowledge-index')).rejects.toThrow('超过 1 页限制') + expect(recognize).not.toHaveBeenCalled() + }) + + it('reports the model error for image-only PPTX, not empty document text', async () => { + const { service } = createService({ modelStatus: { + available: false, verified: false, detail: '模型尚未安装' + } }) + await expect(service.parse('slides.pptx', createImagePptx(), 'chat-attachment')).rejects.toThrow('模型尚未安装') + }) + + it('only falls back to PPTX native text for automatic non-index workflows', async () => { + const { service } = createService({ recognize: async () => { throw new Error('OCR runtime failed') } }) + const bytes = createImagePptx('Useful native slide body') + expect((await service.parse('slides.pptx', bytes, 'chat-attachment')).warnings).toEqual([ + expect.stringContaining('OCR runtime failed') + ]) + await expect(service.parse('slides.pptx', bytes, 'knowledge-index')).rejects.toThrow('OCR runtime failed') + }) + + it('stops PPTX OCR between images on cancellation instead of returning native text', async () => { + const controller = new AbortController() + const { service, recognize } = createService({ recognize: async () => { + controller.abort(new Error('Cancelled PPTX')) + throw new Error('OCR stopped') + } }) + await expect(service.parse('slides.pptx', createImagePptx('Useful native text', 2), 'chat-attachment', controller.signal)).rejects.toThrow('Cancelled PPTX') + expect(recognize).toHaveBeenCalledOnce() + }) + it('keeps useful PDF text local without invoking OCR', async () => { const { recognize, service } = createService() diff --git a/src/main/document-parsing-service.ts b/src/main/document-parsing-service.ts index 3d90e22c..641fe92e 100644 --- a/src/main/document-parsing-service.ts +++ b/src/main/document-parsing-service.ts @@ -13,6 +13,7 @@ import { import type { DocumentOcrBroker } from './document-ocr-broker' import type { DocumentOcrModelManager } from './document-ocr-model-manager' import type { DocumentParsingSettingsStore } from './document-parsing-settings-store' +import { extractPptxPages } from './knowledge/pptx-parser' import { assertDocumentBuffer, DocumentTextUnavailableError, @@ -78,11 +79,12 @@ function effectiveOcrMode( return 'auto' } -function buildPdfDocument( +function buildPagedDocument( name: string, sections: ParsedSection[], pageCount: number, - warnings: string[] = [] + warnings: string[] = [], + sourceFormat: '.pdf' | '.pptx' = '.pdf' ): ParsedDocument { const truncationWarning = '文档提取文本超过 5,000,000 字符，已截断' @@ -130,7 +132,7 @@ function buildPdfDocument( : boundedWarnings.slice(0, maximumDocumentParsingWarnings) return { title: name.replace(/\.[^.]+$/u, ''), - sourceFormat: '.pdf', + sourceFormat, content, sections: limitedSections, pageCount, @@ -204,6 +206,9 @@ export class DocumentParsingService { ) => { ensureNotAborted(signal) assertDocumentBuffer(buffer) + if (extname(name).toLowerCase() === '.pptx') { + return this.parsePptx(name, buffer, purpose, signal) + } if (extname(name).toLowerCase() !== '.pdf') { return parseDocument(name, buffer, signal) } @@ -234,7 +239,7 @@ export class DocumentParsingService { ? ['文档提取文本超过 5,000,000 字符，已截断'] : []) ] - return buildPdfDocument( + return buildPagedDocument( name, native, extracted.pageCount, @@ -246,7 +251,7 @@ export class DocumentParsingService { ) } if (ocrPageNumbers.length === 0) { - return buildPdfDocument( + return buildPagedDocument( name, nativePdfSections(pages), extracted.pageCount, @@ -270,7 +275,7 @@ export class DocumentParsingService { purpose !== 'knowledge-index' && native.some((section) => hasUsefulText(section.content)) ) { - return buildPdfDocument( + return buildPagedDocument( name, native, extracted.pageCount, @@ -308,7 +313,7 @@ export class DocumentParsingService { ) { const detail = error instanceof Error ? error.message : '本地 OCR 识别失败' - return buildPdfDocument( + return buildPagedDocument( name, native, extracted.pageCount, @@ -367,7 +372,7 @@ export class DocumentParsingService { }] : [] }) - return buildPdfDocument( + return buildPagedDocument( name, merged, extracted.pageCount, @@ -380,6 +385,115 @@ export class DocumentParsingService { ) } + private async parsePptx( + name: string, + buffer: Buffer, + purpose: DocumentParsingPurpose, + signal?: AbortSignal + ): Promise { + const settings = await this.settingsStore.get() + const mode = effectiveOcrMode(settings, purpose) + if (mode === 'disabled') { + try { + return await parseDocument(name, buffer, signal) + } catch (error) { + if (error instanceof DocumentTextUnavailableError) { + throw new DocumentTextUnavailableError( + 'PPTX 没有可用文本，当前工作流未启用 OCR' + ) + } + throw error + } + } + const pages = extractPptxPages(buffer) + ensureNotAborted(signal) + const native: ParsedSection[] = pages + .filter((page) => page.content) + .map((page) => ({ + locator: `幻灯片 ${page.pageNumber}`, + content: page.content, + method: 'native', + pageNumber: page.pageNumber, + blockKind: 'slide' + })) + const imagePages = pages.filter((page) => page.images.length > 0) + if (imagePages.length > settings.maximumPages) { + throw new Error( + `PPTX 有 ${imagePages.length} 页需要 OCR，超过 ${settings.maximumPages} 页限制` + ) + } + const sections = [...native] + const warnings: string[] = [] + let extractedCharacters = native.reduce( + (total, section) => total + section.content.length, 0 + ) + if (imagePages.length > 0) { + try { + const status = await this.modelManager.getStatus(settings.localOcrModelId) + if (!status.available || !status.verified) { + throw new Error(status.detail) + } + ocrPages: for (const page of imagePages) { + for (const [index, image] of page.images.entries()) { + ensureNotAborted(signal) + const locator = `幻灯片 ${page.pageNumber} · 图片 ${index + 1}` + if (!image.mimeType) { + throw new Error(`${locator} 的格式暂不支持 OCR：${extname(image.name)}`) + } + const request = { + modelId: settings.localOcrModelId, + fileName: image.name, + mimeType: image.mimeType, + data: Uint8Array.from(image.data).buffer, + maximumPages: 1, + pageTimeoutSeconds: settings.pageTimeoutSeconds + } + const result = await (signal + ? this.ocrBroker.recognize(request, signal) + : this.ocrBroker.recognize(request)) + ensureNotAborted(signal) + sections.push(...result.sections.map((section): ParsedSection => ({ + locator, + content: section.content, + confidence: section.confidence, + method: 'ocr', + pageNumber: page.pageNumber, + blockKind: 'slide' + }))) + for (const warning of result.warnings) { + if (warnings.length < maximumDocumentParsingWarnings) { + warnings.push(`${locator}：${warning}`.slice(0, 500)) + } + } + extractedCharacters += result.sections.reduce( + (total, section) => total + section.content.length, 0 + ) + if (extractedCharacters >= maximumDocumentExtractedCharacters) { + warnings.push('文档提取文本超过 5,000,000 字符，已截断') + break ocrPages + } + } + } + } catch (error) { + ensureNotAborted(signal) + if ( + mode !== 'auto' || + purpose === 'knowledge-index' || + !native.some((section) => hasUsefulText(section.content)) + ) { + throw error + } + return buildPagedDocument(name, native, pages.length, [ + `本地 OCR 不可用，已保留 PPTX 文本内容：${ + error instanceof Error ? error.message : '本地 OCR 识别失败' + }`.slice(0, 500) + ], '.pptx') + } + } + sections.sort((left, right) => (left.pageNumber ?? 0) - (right.pageNumber ?? 0)) + return buildPagedDocument(name, sections, pages.length, warnings, '.pptx') + } + async diagnose( name: string, buffer: Buffer, @@ -387,9 +501,9 @@ export class DocumentParsingService { ): Promise { const startedAt = Date.now() const parsed = await this.parse(name, buffer, purpose) - const ocrPageCount = parsed.sections.filter( + const ocrPageCount = new Set(parsed.sections.filter( (section) => section.method === 'ocr' - ).length + ).map((section) => section.pageNumber ?? section.locator)).size const nativePageCount = parsed.sections.filter( (section) => section.method !== 'ocr' ).length @@ -398,7 +512,7 @@ export class DocumentParsingService { sourceFormat: parsed.sourceFormat.replace(/^\./u, '').toUpperCase() || 'UNKNOWN', pageCount: - parsed.sourceFormat === '.pdf' + parsed.sourceFormat === '.pdf' || parsed.sourceFormat === '.pptx' ? (parsed.pageCount ?? parsed.sections.length) : 0, ocrPageCount, diff --git a/src/main/knowledge/document-parser.ts b/src/main/knowledge/document-parser.ts index 3ce3834f..c01c05bf 100644 --- a/src/main/knowledge/document-parser.ts +++ b/src/main/knowledge/document-parser.ts @@ -102,7 +102,7 @@ function decodeXmlEntities(value: string): string { ) } -function extractXmlText(xml: string): string { +export function extractXmlText(xml: string): string { return decodeXmlEntities( xml .replace(/]*\/>/g, '\t') @@ -239,19 +239,10 @@ function extractWorksheetText( .trim() } -function parseOfficeArchive( +export function readOfficeArchive( buffer: Buffer, - extension: string -): ParsedSection[] { - const patterns = - extension === '.docx' - ? [/^word\/document\.xml$/] - : extension === '.xlsx' - ? [ - /^xl\/sharedStrings\.xml$/, - /^xl\/worksheets\/sheet\d+\.xml$/ - ] - : [/^ppt\/slides\/slide\d+\.xml$/] + patterns: RegExp[] +): Record { let archive: Record let entryCount = 0 let selectedBytes = 0 @@ -281,6 +272,24 @@ function parseOfficeArchive( } catch { throw new Error('Office 文档已损坏或不是有效的 Open XML 文件') } + return archive +} + +function parseOfficeArchive( + buffer: Buffer, + extension: string +): ParsedSection[] { + const archive = readOfficeArchive( + buffer, + extension === '.docx' + ? [/^word\/document\.xml$/] + : extension === '.xlsx' + ? [ + /^xl\/sharedStrings\.xml$/, + /^xl\/worksheets\/sheet\d+\.xml$/ + ] + : [/^ppt\/slides\/slide\d+\.xml$/] + ) if (extension === '.docx') { const documentXml = archive['word/document.xml'] diff --git a/src/main/knowledge/pptx-parser.test.ts b/src/main/knowledge/pptx-parser.test.ts new file mode 100644 index 00000000..df7e8a75 --- /dev/null +++ b/src/main/knowledge/pptx-parser.test.ts @@ -0,0 +1,37 @@ +import { strToU8, zipSync } from 'fflate' +import { describe, expect, it } from 'vitest' +import { extractPptxPages } from './pptx-parser' +import { createImagePptx } from '../../../tests/support/pptx-fixture' + +describe('PPTX image extraction', () => { + it('extracts referenced images with slide locators', () => { + const pages = extractPptxPages(createImagePptx('Native text', 2)) + expect(pages.map((page) => page.pageNumber)).toEqual([1, 2]) + expect(pages[0]).toEqual({ + pageNumber: 1, + content: 'Native text', + images: [{ + name: 'image1.png', + data: new Uint8Array([1, 2, 3]), + mimeType: 'image/png' + }] + }) + }) + + it('follows presentation order and ignores external and unreferenced media', () => { + const pages = extractPptxPages(Buffer.from(zipSync({ + 'ppt/presentation.xml': strToU8(''), + 'ppt/_rels/presentation.xml.rels': strToU8(''), + 'ppt/slides/slide1.xml': strToU8('First'), + 'ppt/slides/_rels/slide1.xml.rels': strToU8(''), + 'ppt/slides/slide2.xml': strToU8('Second'), + 'ppt/media/unused.png': new Uint8Array([1]) + }))) + expect(pages.map(({ pageNumber, content, images }) => ({ + pageNumber, content, images + }))).toEqual([ + { pageNumber: 1, content: 'Second', images: [] }, + { pageNumber: 2, content: 'First', images: [] } + ]) + }) +}) diff --git a/src/main/knowledge/pptx-parser.ts b/src/main/knowledge/pptx-parser.ts new file mode 100644 index 00000000..528b7cd8 --- /dev/null +++ b/src/main/knowledge/pptx-parser.ts @@ -0,0 +1,99 @@ +import { posix } from 'node:path' +import { extractXmlText, readOfficeArchive } from './document-parser' + +export type PptxPage = { + pageNumber: number + content: string + images: Array<{ + name: string + data: Uint8Array + mimeType?: 'image/png' | 'image/jpeg' | 'image/webp' + }> +} + +function attribute(tag: string, name: string): string | undefined { + return new RegExp(`\\b${name}=["']([^"']*)["']`, 'u').exec(tag)?.[1] +} + +function relationships(xml: string, owner: string): Map { + const result = new Map() + for (const tag of xml.match(/<(?:\w+:)?Relationship\b[^>]*>/gu) ?? []) { + const id = attribute(tag, 'Id') + const target = attribute(tag, 'Target') + if (!id || !target || attribute(tag, 'TargetMode') === 'External') { + continue + } + result.set( + id, + posix.normalize( + target.startsWith('/') + ? target.slice(1) + : posix.join(posix.dirname(owner), target) + ) + ) + } + return result +} + +export function extractPptxPages(buffer: Buffer): PptxPage[] { + const archive = readOfficeArchive(buffer, [ + /^ppt\/presentation\.xml$/, + /^ppt\/_rels\/presentation\.xml\.rels$/, + /^ppt\/slides\/[^/]+\.xml$/, + /^ppt\/slides\/_rels\/[^/]+\.xml\.rels$/, + /^ppt\/media\/[^/]+$/ + ]) + const text = (name: string): string => + archive[name] ? Buffer.from(archive[name]).toString('utf8') : '' + const slideRelationships = relationships( + text('ppt/_rels/presentation.xml.rels'), + 'ppt/presentation.xml' + ) + const slideIds = + text('ppt/presentation.xml').match(/<(?:\w+:)?sldId\b[^>]*>/gu) ?? [] + const paths = slideIds.length > 0 + ? slideIds.map((tag) => { + const path = slideRelationships.get(attribute(tag, 'r:id') ?? '') + if (!path || !archive[path]) { + throw new Error('PPTX 幻灯片关系无效') + } + return path + }) + : Object.keys(archive) + .filter((path) => /^ppt\/slides\/slide\d+\.xml$/u.test(path)) + .sort((left, right) => left.localeCompare(right, undefined, { numeric: true })) + + return paths.map((path, index) => { + const xml = text(path) + const imageRelationships = relationships( + text(posix.join(posix.dirname(path), '_rels', `${posix.basename(path)}.rels`)), + path + ) + const imagePaths = new Set( + (xml.match(/<(?:\w+:)?blip\b[^>]*>/gu) ?? []) + .map((tag) => imageRelationships.get(attribute(tag, 'r:embed') ?? '')) + .filter((value): value is string => value !== undefined) + ) + const images: PptxPage['images'] = [] + for (const imagePath of imagePaths) { + const extension = posix.extname(imagePath).toLowerCase() + const mimeType = extension === '.png' + ? 'image/png' + : extension === '.jpg' || extension === '.jpeg' + ? 'image/jpeg' + : extension === '.webp' + ? 'image/webp' + : undefined + const data = archive[imagePath] + if (!data) { + throw new Error(`幻灯片 ${index + 1} 的内嵌图片缺失`) + } + images.push({ name: posix.basename(imagePath), data, mimeType }) + } + return { + pageNumber: index + 1, + content: extractXmlText(xml), + images + } + }) +} diff --git a/src/renderer/src/document-ocr-bridge.test.ts b/src/renderer/src/document-ocr-bridge.test.ts new file mode 100644 index 00000000..b6db4479 --- /dev/null +++ b/src/renderer/src/document-ocr-bridge.test.ts @@ -0,0 +1,156 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import type { DocumentOcrRequest } from '../../shared/document-parsing-contracts' + +class TestWorker extends EventTarget { + static instances: TestWorker[] = [] + terminate = vi.fn() + postMessage = vi.fn((message: { type: string }) => { + if (message.type === 'initialize') { + this.dispatchEvent(new MessageEvent('message', { data: { type: 'ready' } })) + } + }) + + constructor() { + super() + TestWorker.instances.push(this) + } + + finish(requestId: string): void { + this.dispatchEvent(new MessageEvent('message', { + data: { type: 'result', result: { requestId, sections: [], pageCount: 1, warnings: [] } } + })) + } +} + +function request(): DocumentOcrRequest { + return { + requestId: crypto.randomUUID(), + modelId: 'pp-ocrv6-tiny', + fileName: 'scan.png', + mimeType: 'image/png', + data: new ArrayBuffer(1), + maximumPages: 1, + pageTimeoutSeconds: 300 + } +} + +let dispose: (() => void) | undefined + +beforeEach(() => { + vi.resetModules() + vi.useFakeTimers() + TestWorker.instances = [] + vi.stubGlobal('Worker', TestWorker) +}) + +afterEach(async () => { + dispose?.() + dispose = undefined + await vi.advanceTimersByTimeAsync(0) + vi.useRealTimers() + vi.unstubAllGlobals() +}) + +async function setup() { + let receive!: (request: DocumentOcrRequest) => void + let cancel!: (requestId: string) => void + const getOcrAssets = vi.fn(async (modelId: string) => ({ + modelId, detection: new ArrayBuffer(1), recognition: new ArrayBuffer(1), dictionary: new ArrayBuffer(1) + })) + const respondOcr = vi.fn(async () => undefined) + const removeRequest = vi.fn() + const removeCancel = vi.fn() + vi.stubGlobal('goodbuddy', { + documentParsing: { + getOcrAssets, respondOcr, + onOcrRequest: (listener: typeof receive) => { receive = listener; return removeRequest }, + onOcrCancel: (listener: typeof cancel) => { cancel = listener; return removeCancel } + } + }) + const { installDocumentOcrBridge } = await import('./document-ocr-bridge') + dispose = installDocumentOcrBridge() + const start = async () => { + const input = request() + receive(input) + await vi.advanceTimersByTimeAsync(0) + return input + } + const finish = async (input: DocumentOcrRequest) => { + TestWorker.instances.at(-1)!.finish(input.requestId) + await vi.advanceTimersByTimeAsync(0) + } + return { start, finish, receive, cancel, getOcrAssets, respondOcr, removeRequest, removeCancel } +} + +describe('document OCR worker idle release', () => { + it('releases after 60 idle seconds and reloads the model for the next request', async () => { + const api = await setup() + const input = await api.start() + const worker = TestWorker.instances[0]! + await api.finish(input) + await vi.advanceTimersByTimeAsync(59_999) + expect(worker.terminate).not.toHaveBeenCalled() + await vi.advanceTimersByTimeAsync(1) + expect(worker.terminate).toHaveBeenCalledOnce() + const next = await api.start() + expect(TestWorker.instances).toHaveLength(2) + expect(api.getOcrAssets).toHaveBeenCalledTimes(2) + await api.finish(next) + }) + + it('cancels the idle timer for new work and reuses the loaded worker', async () => { + const api = await setup() + await api.finish(await api.start()) + await vi.advanceTimersByTimeAsync(59_000) + const next = await api.start() + const worker = TestWorker.instances[0]! + await vi.advanceTimersByTimeAsync(61_000) + expect(worker.terminate).not.toHaveBeenCalled() + expect(api.getOcrAssets).toHaveBeenCalledOnce() + await api.finish(next) + await vi.advanceTimersByTimeAsync(60_000) + expect(worker.terminate).toHaveBeenCalledOnce() + }) + + it('does not release while a second request is queued or running', async () => { + const api = await setup() + const first = await api.start() + const second = await api.start() + const worker = TestWorker.instances[0]! + await vi.advanceTimersByTimeAsync(61_000) + expect(worker.terminate).not.toHaveBeenCalled() + await api.finish(first) + await vi.advanceTimersByTimeAsync(61_000) + expect(worker.terminate).not.toHaveBeenCalled() + await api.finish(second) + await vi.advanceTimersByTimeAsync(60_000) + expect(worker.terminate).toHaveBeenCalledOnce() + expect(api.respondOcr).toHaveBeenCalledTimes(2) + }) + + it('clears the idle timer on disposal and removes subscriptions', async () => { + const api = await setup() + await api.finish(await api.start()) + const worker = TestWorker.instances[0]! + dispose?.() + dispose = undefined + expect(api.removeRequest).toHaveBeenCalledOnce() + expect(api.removeCancel).toHaveBeenCalledOnce() + await vi.advanceTimersByTimeAsync(60_000) + expect(worker.terminate).toHaveBeenCalledOnce() + }) + + it('still terminates immediately on cancellation', async () => { + const api = await setup() + const input = await api.start() + const worker = TestWorker.instances[0]! + api.cancel(input.requestId) + await vi.advanceTimersByTimeAsync(0) + expect(worker.terminate).toHaveBeenCalledOnce() + expect(api.respondOcr).toHaveBeenCalledWith({ + requestId: input.requestId, error: '本地 OCR 解析已取消' + }) + await vi.advanceTimersByTimeAsync(60_000) + expect(worker.terminate).toHaveBeenCalledOnce() + }) +}) diff --git a/src/renderer/src/document-ocr-bridge.ts b/src/renderer/src/document-ocr-bridge.ts index 5f4d0a77..f15587a1 100644 --- a/src/renderer/src/document-ocr-bridge.ts +++ b/src/renderer/src/document-ocr-bridge.ts @@ -26,6 +26,26 @@ const pending = new Map() const cancelledRequestIds = new Set() let activeRequestId: string | undefined let requestQueue: Promise = Promise.resolve() +let queuedRequestCount = 0 +let idleTimer: number | undefined +const workerIdleTimeoutMs = 60_000 + +function clearIdleTimer(): void { + window.clearTimeout(idleTimer) + idleTimer = undefined +} + +function scheduleIdleRelease(): void { + clearIdleTimer() + if (!worker || queuedRequestCount > 0 || activeRequestId || pending.size > 0) { + return + } + idleTimer = window.setTimeout(() => { + if (queuedRequestCount === 0 && !activeRequestId && pending.size === 0) { + terminateWorker(new Error('本地 OCR 空闲资源已释放')) + } + }, workerIdleTimeoutMs) +} function safeError(error: unknown): string { return error instanceof Error @@ -34,6 +54,7 @@ function safeError(error: unknown): string { } function terminateWorker(error: Error): void { + clearIdleTimer() worker?.terminate() rejectWorkerReady?.(error) worker = undefined @@ -199,9 +220,15 @@ export function installDocumentOcrBridge(): () => void { return () => undefined } const removeRequestListener = api.onOcrRequest((request) => { + clearIdleTimer() + queuedRequestCount += 1 requestQueue = requestQueue .then(() => handleRequest(request)) .catch(() => undefined) + .finally(() => { + queuedRequestCount -= 1 + scheduleIdleRelease() + }) }) const removeCancelListener = api.onOcrCancel((requestId) => { cancelledRequestIds.add(requestId) diff --git a/tests/support/pptx-fixture.ts b/tests/support/pptx-fixture.ts new file mode 100644 index 00000000..b87c21e5 --- /dev/null +++ b/tests/support/pptx-fixture.ts @@ -0,0 +1,16 @@ +import { strToU8, zipSync } from 'fflate' + +export function createImagePptx(nativeText = '', pageCount = 1): Buffer { + const files: Record = { + 'ppt/media/image1.png': new Uint8Array([1, 2, 3]) + } + for (let page = 1; page <= pageCount; page += 1) { + files[`ppt/slides/slide${page}.xml`] = strToU8( + `${nativeText}` + ) + files[`ppt/slides/_rels/slide${page}.xml.rels`] = strToU8( + '' + ) + } + return Buffer.from(zipSync(files)) +}