fix: recognize PPTX images and release idle OCR memory
Image-only PPTX attachments failed with no indexable text even when a local OCR model was configured. PPTX parsing now recognizes embedded PNG, JPEG and WebP images through the existing local OCR path, preserves native text and slide locators, and honors the selected parsing workflow. OCR workers previously retained model sessions and WASM memory indefinitely after successful parsing. Release the worker after 60 seconds with no active or queued requests, reuse it for continuous work, and reload on demand. Fast text/index modes still skip OCR. This does not render Office layouts, recognize inherited master images, or change Agent execution permissions. Validated four real PPTX attachments (40 pages), idle memory release and a nine-page reimport after model reload on Windows x64 with PP-OCRv6 Tiny. 49 local OCR requests and zero text-model calls. All 4293 tests passed with 67 skipped; typecheck, lint and the isolated Electron build passed. Release note: 修复已配置本地 OCR 时图片型 PPTX 仍无法导入的问题,识别结果保留幻灯片定位。OCR 空闲 60 秒后自动释放模型运行内存,连续解析复用模型,下次使用时自动重新加载。
This commit is contained in:
@@ -7,7 +7,7 @@
|
||||
| 文档 | 权威职责 |
|
||||
| --- | --- |
|
||||
| [文档解析与本地 OCR PRD](./prd.md) | 产品范围、解析策略、OCR 模型管理和验收 |
|
||||
| [聊天附件技术说明](./chat-attachments-technical-design.md) | 文件选择与粘贴的进程链路、IPC、失败处理和验证方法 |
|
||||
| [聊天附件技术说明](./chat-attachments-technical-design.md) | 文件选择与粘贴的进程链路、IPC、PPTX 图片 OCR、Worker 生命周期和验证证据 |
|
||||
|
||||
当前尚无独立 User Stories、功能逻辑、UI 和进度文档;聊天输入区交互遵循根目录
|
||||
[UI 设计系统](../../../UI-DESIGN.md)。
|
||||
|
||||
@@ -24,6 +24,27 @@ Main。IPC 先验证可信 sender,再使用共享 `contextImportFilesSchema`
|
||||
会话草稿;发送时通过已有 `contextIds` 和 `ContextManager.enrichRequest` 生成请求上下文。
|
||||
本次变更不修改 Agent、远端协议或 Runtime 的附件消费实现。
|
||||
|
||||
## PPTX 图片识别与 OCR 生命周期
|
||||
|
||||
产品策略与限制分别见 [PPTX 图片流程](./prd.md#75-pptx-图片流程)、
|
||||
[OCR 空闲资源回收](./prd.md#76-ocr-空闲资源回收)。
|
||||
|
||||
`DocumentParsingService` 在非快速工作流中使用 `knowledge/pptx-parser.ts` 读取
|
||||
演示文稿顺序、幻灯片文字、图片关系和内嵌图片。Office ZIP 读取复用
|
||||
`document-parser.ts` 的条目数量、单条目及总展开大小检查;不读取工作区中的关联文件。
|
||||
每张引用图片通过现有 `DocumentOcrBroker` 的图片请求契约,进入生产
|
||||
`document-ocr-bridge.ts` 和 WASM Worker。结果合并时保留原生文字,附上幻灯片页码、
|
||||
图片定位及 OCR 置信度,诊断中的 OCR 页数按页码去重。
|
||||
|
||||
Bridge 用待完成请求计数判断队列是否空闲,正常完成后安排一次空闲计时;
|
||||
新请求和 `terminateWorker` 均清除计时。到期通过 `Worker.terminate()` 释放整个推理环境,
|
||||
而不是只清空 JavaScript 引用或反复触发垃圾回收。此机制由 PDF 与 PPTX 共用,
|
||||
不增加 IPC、持久化设置或模型安装状态。
|
||||
|
||||
解析与回收发生在 Desktop 的 Main/Renderer,`gbagent` 不运行此 OCR Worker。
|
||||
远程 Runtime 消费的仍是 Desktop 已提取的附件文本,没有单独的 Agent 解析器或
|
||||
需要同步修改的远程 OCR 生命周期。
|
||||
|
||||
## Electron 验证
|
||||
|
||||
2026-09-15 在 Windows 的 Electron 43.2.0 上,从当前源码构建隔离输出并启动完整应用,
|
||||
@@ -67,3 +88,28 @@ npm run lint
|
||||
沙箱用例,类型检查与 lint 通过。完整 `npm test` 运行结果为 4214 项通过、67 项跳过、
|
||||
6 项窗口恢复用例失败;这些用例属于并行修改的 `window.test.ts`,随后该文件单独复跑
|
||||
10 项全部通过。本次没有再次完整重跑全仓测试。
|
||||
|
||||
## 2026-09-15 PPTX OCR 与空闲回收验证
|
||||
|
||||
在 Windows x64、Electron 43.2.0 上,从本次源码构建独立输出,使用独立用户数据目录和
|
||||
本机已安装的 PP-OCRv6 Tiny 模型完成真实验证。测试仅替换系统文件选择对话框的返回路径,
|
||||
保留生产 UI、preload、IPC、`ContextManager`、解析服务、OCR Broker 与 WASM Worker,
|
||||
未替换解析或推理实现。
|
||||
|
||||
- 从“添加附件”一次导入 4 份图片型 PPTX,共 40 页;4 个附件均成功显示,连续处理期间
|
||||
仅创建 1 个 OCR Worker。
|
||||
- 队列空闲后观察到 Worker 被终止。每 5 秒读取 Electron `app.getAppMetrics()`:
|
||||
Renderer 工作集采样高点为 733,492 KiB(约 716 MiB),回收后为 162,716 KiB
|
||||
(约 159 MiB),启动基线约 163 MiB。该结果是本次设备与文件上的测量,不是固定内存承诺。
|
||||
- 回收后,通过生产 `context.importFiles` 再导入一份 9 页 PPTX;新建第 2 个 Worker,
|
||||
9 页均返回非空文字,共 2,462 字符,附件成功生成。
|
||||
- 本次合计 49 次本地图片 OCR 请求,文本模型调用 0 次;没有发送聊天消息,也没有将文件
|
||||
交给云端模型。未修改源文件。未在 macOS、Linux 或其他 OCR 模型档位上做本次实测。
|
||||
|
||||
新增聚焦回归覆盖 PPTX 关系与顺序、各场景路由、原生文字合并、快速模式、页数上限、
|
||||
OCR 错误和取消;Worker 回归覆盖空闲到期、连续请求复用、队列保护、关闭清理与立即取消。
|
||||
空闲回收后重新加载也在真实模型路径上验证,不能只用模型配置检查代替。
|
||||
|
||||
最终检查:`npm test` 4,293 项通过、67 项跳过;`npm run typecheck`、
|
||||
`npm run lint` 通过。当前源码的隔离 Electron 生产构建通过;文档的 10 个相对文件链接
|
||||
均有效。测试应用已关闭,包含附件草稿的独立用户数据目录已删除,原文件和原模型安装未改动。
|
||||
|
||||
@@ -22,13 +22,16 @@ GoodBuddy 需要用同一条可信文档解析链路服务以下场景:
|
||||
- XLSX 工作表 XML 与共享字符串;
|
||||
- PPTX 幻灯片文字。
|
||||
|
||||
聊天附件、知识库和测试解析通过统一的 `DocumentParsingService` 使用已保存的工作流。
|
||||
扫描 PDF 和 PPTX 幻灯片内嵌图片可使用已安装、已校验的本地 OCR 模型识别;
|
||||
PPTX 的当前能力和边界见 [PPTX 图片流程](#75-pptx-图片流程)。
|
||||
|
||||
现有局限:
|
||||
|
||||
- 纯扫描 PDF 没有文本层时无法提取内容;
|
||||
- DOC、XLS、PPT 等旧版二进制 Office 格式不支持;
|
||||
- Office 解析主要提取文字,不能完整保留表格、公式、图表和版面;
|
||||
- 聊天附件和知识库直接调用底层解析函数,缺少可配置的统一工作流;
|
||||
- 没有本地 OCR 模型状态、真实解析测试和按场景策略。
|
||||
- DOCX、XLSX 内嵌图片尚未接入 OCR;
|
||||
- Office 格式转换和远程 OCR 服务尚未接入。
|
||||
|
||||
## 3. 产品原则
|
||||
|
||||
@@ -277,6 +280,32 @@ ZIP 根目录包含模型文件和 `goodbuddy-model.json`。清单格式为 `goo
|
||||
|
||||
受密码保护、损坏或超限的 PDF 不得进入 OCR。
|
||||
|
||||
### 7.5 PPTX 图片流程
|
||||
|
||||
自动解析、完整索引和高保真工作流保留幻灯片原生文字,同时对幻灯片直接引用的
|
||||
内嵌 PNG、JPEG、WebP 图片运行本地 OCR。即使幻灯片已有标题或正文,也不会因此
|
||||
跳过图片中的文字。快速文本、快速索引工作流不运行 OCR;图片型 PPTX 在这些模式下
|
||||
明确提示当前工作流未启用 OCR。
|
||||
|
||||
按演示文稿中的幻灯片顺序返回结果,图片识别结果标注 `幻灯片 N · 图片 M`,
|
||||
保留页码和识别置信度。OCR 页数限制计算含图片的幻灯片数,而不是图片总数;
|
||||
图片顺序处理并复用已加载模型,沿用现有超时、取消和输出限制。
|
||||
|
||||
这不是 Office 渲染或转换:不识别母版、版式中继承的图片,不重建裁剪、旋转或遮挡后的
|
||||
完整画面,不把演讲者备注当作幻灯片正文,也不下载外链图片。其他内嵌图片格式在
|
||||
需要 OCR 时明确报错。自动聊天解析可在 OCR 失败时保留可靠原生文字并附带警告;
|
||||
图片型文件没有可靠原生文字时仍失败,知识库和高保真工作流不以此回退掩盖 OCR 失败。
|
||||
|
||||
### 7.6 OCR 空闲资源回收
|
||||
|
||||
OCR 队列清空且没有正在处理的请求后,保留 Worker 60 秒,以复用连续文档的模型。
|
||||
期间收到新请求立即取消回收计时;请求完成、队列再次空闲后重新计时。空闲到期后
|
||||
销毁 Worker,释放其模型会话和 WASM 运行环境;下一次请求按需重新加载已安装模型。
|
||||
|
||||
回收不会删除模型文件或已经解析的附件文字,不打断正在识别、初始化或排队的任务。
|
||||
取消、超时和服务关闭仍按原有路径立即清理。操作系统显示的进程内存不保证立刻恢复到
|
||||
启动值,不能仅凭一次内存快照判断是否发生泄漏。
|
||||
|
||||
## 8. Office 与转换
|
||||
|
||||
本节只定义 Office 文件的语义解析和格式转换,不提供可视化编辑器、编辑会话、选区、撤销或
|
||||
@@ -361,6 +390,7 @@ DOC、XLS、PPT 通过 `DocumentConversionProvider` 转换:
|
||||
## 12. 验收
|
||||
|
||||
- 同一份扫描 PDF 可从聊天附件和知识库得到一致的逐页文本;
|
||||
- 图片型 PPTX 在自动解析下使用已配置的本地 OCR,并保留幻灯片来源定位;
|
||||
- 文本型 PDF 在 `auto` 模式下不运行 OCR;
|
||||
- 本地 OCR 在六个平台和两种架构上完全离线运行;
|
||||
- 模型文件损坏时拒绝加载并显示可恢复错误;
|
||||
@@ -371,6 +401,7 @@ DOC、XLS、PPT 通过 `DocumentConversionProvider` 转换:
|
||||
- 语音和 OCR 模型可在联网设备导出 ZIP,并在离线设备导入后完成真实推理;
|
||||
- 路径穿越、未知条目、错误模型 ID、篡改文件和超限 ZIP 均被拒绝;
|
||||
- 超页数、超时、取消和关闭不会留下运行任务;
|
||||
- OCR 空闲 60 秒后释放 Worker,连续请求复用模型,回收后的下一次解析可重新加载模型完成;
|
||||
- 测试解析不会创建聊天消息或知识库文档;
|
||||
- 选择本地模型时没有任何文档上传;
|
||||
- 文档中的提示词不会改变系统、模式或工具权限。
|
||||
|
||||
@@ -3,6 +3,7 @@ import {
|
||||
defaultDocumentParsingSettings
|
||||
} from './document-parsing-settings-store'
|
||||
import { DocumentParsingService } from './document-parsing-service'
|
||||
import { createImagePptx } from '../../tests/support/pptx-fixture'
|
||||
|
||||
function createPdfFixture(...pageTexts: string[]): Buffer {
|
||||
const texts = pageTexts.length > 0 ? pageTexts : ['']
|
||||
@@ -109,6 +110,73 @@ function createService(overrides?: {
|
||||
}
|
||||
|
||||
describe('DocumentParsingService', () => {
|
||||
it.each(['chat-attachment', 'knowledge-index', 'artifact-import', 'diagnostic'] as const)(
|
||||
'recognizes embedded PPTX images for %s',
|
||||
async (purpose) => {
|
||||
const { recognize, service } = createService()
|
||||
const parsed = await service.parse('slides.pptx', createImagePptx('', 2), purpose)
|
||||
expect(parsed.sourceFormat).toBe('.pptx')
|
||||
expect(parsed.pageCount).toBe(2)
|
||||
expect(parsed.sections.map(({ locator, method, pageNumber }) => ({
|
||||
locator, method, pageNumber
|
||||
}))).toEqual([
|
||||
{ locator: '幻灯片 1 · 图片 1', method: 'ocr', pageNumber: 1 },
|
||||
{ locator: '幻灯片 2 · 图片 1', method: 'ocr', pageNumber: 2 }
|
||||
])
|
||||
expect(recognize).toHaveBeenCalledTimes(2)
|
||||
expect(recognize).toHaveBeenCalledWith(expect.objectContaining({
|
||||
mimeType: 'image/png', maximumPages: 1, modelId: 'pp-ocrv6-tiny'
|
||||
}))
|
||||
}
|
||||
)
|
||||
|
||||
it('preserves PPTX native text while recognizing image content on the same slide', async () => {
|
||||
const { service } = createService()
|
||||
const parsed = await service.parse('slides.pptx', createImagePptx('Native slide title'), 'chat-attachment')
|
||||
expect(parsed.content).toContain('Native slide title')
|
||||
expect(parsed.content).toContain('扫描件识别正文')
|
||||
const diagnostic = await service.diagnose('slides.pptx', createImagePptx('Native slide title'))
|
||||
expect(diagnostic).toMatchObject({ method: 'mixed', pageCount: 1, ocrPageCount: 1 })
|
||||
})
|
||||
|
||||
it('reports disabled OCR for image-only PPTX in fast-text mode', async () => {
|
||||
const { service, recognize } = createService({ settings: { chatWorkflow: 'fast-text' } })
|
||||
await expect(service.parse('slides.pptx', createImagePptx(), 'chat-attachment')).rejects.toThrow('当前工作流未启用 OCR')
|
||||
expect(recognize).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('enforces PPTX OCR page limits before recognizing images', async () => {
|
||||
const { service, recognize } = createService({ settings: { maximumPages: 1 } })
|
||||
await expect(service.parse('slides.pptx', createImagePptx('', 2), 'knowledge-index')).rejects.toThrow('超过 1 页限制')
|
||||
expect(recognize).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('reports the model error for image-only PPTX, not empty document text', async () => {
|
||||
const { service } = createService({ modelStatus: {
|
||||
available: false, verified: false, detail: '模型尚未安装'
|
||||
} })
|
||||
await expect(service.parse('slides.pptx', createImagePptx(), 'chat-attachment')).rejects.toThrow('模型尚未安装')
|
||||
})
|
||||
|
||||
it('only falls back to PPTX native text for automatic non-index workflows', async () => {
|
||||
const { service } = createService({ recognize: async () => { throw new Error('OCR runtime failed') } })
|
||||
const bytes = createImagePptx('Useful native slide body')
|
||||
expect((await service.parse('slides.pptx', bytes, 'chat-attachment')).warnings).toEqual([
|
||||
expect.stringContaining('OCR runtime failed')
|
||||
])
|
||||
await expect(service.parse('slides.pptx', bytes, 'knowledge-index')).rejects.toThrow('OCR runtime failed')
|
||||
})
|
||||
|
||||
it('stops PPTX OCR between images on cancellation instead of returning native text', async () => {
|
||||
const controller = new AbortController()
|
||||
const { service, recognize } = createService({ recognize: async () => {
|
||||
controller.abort(new Error('Cancelled PPTX'))
|
||||
throw new Error('OCR stopped')
|
||||
} })
|
||||
await expect(service.parse('slides.pptx', createImagePptx('Useful native text', 2), 'chat-attachment', controller.signal)).rejects.toThrow('Cancelled PPTX')
|
||||
expect(recognize).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('keeps useful PDF text local without invoking OCR', async () => {
|
||||
const { recognize, service } = createService()
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@ import {
|
||||
import type { DocumentOcrBroker } from './document-ocr-broker'
|
||||
import type { DocumentOcrModelManager } from './document-ocr-model-manager'
|
||||
import type { DocumentParsingSettingsStore } from './document-parsing-settings-store'
|
||||
import { extractPptxPages } from './knowledge/pptx-parser'
|
||||
import {
|
||||
assertDocumentBuffer,
|
||||
DocumentTextUnavailableError,
|
||||
@@ -78,11 +79,12 @@ function effectiveOcrMode(
|
||||
return 'auto'
|
||||
}
|
||||
|
||||
function buildPdfDocument(
|
||||
function buildPagedDocument(
|
||||
name: string,
|
||||
sections: ParsedSection[],
|
||||
pageCount: number,
|
||||
warnings: string[] = []
|
||||
warnings: string[] = [],
|
||||
sourceFormat: '.pdf' | '.pptx' = '.pdf'
|
||||
): ParsedDocument {
|
||||
const truncationWarning =
|
||||
'文档提取文本超过 5,000,000 字符,已截断'
|
||||
@@ -130,7 +132,7 @@ function buildPdfDocument(
|
||||
: boundedWarnings.slice(0, maximumDocumentParsingWarnings)
|
||||
return {
|
||||
title: name.replace(/\.[^.]+$/u, ''),
|
||||
sourceFormat: '.pdf',
|
||||
sourceFormat,
|
||||
content,
|
||||
sections: limitedSections,
|
||||
pageCount,
|
||||
@@ -204,6 +206,9 @@ export class DocumentParsingService {
|
||||
) => {
|
||||
ensureNotAborted(signal)
|
||||
assertDocumentBuffer(buffer)
|
||||
if (extname(name).toLowerCase() === '.pptx') {
|
||||
return this.parsePptx(name, buffer, purpose, signal)
|
||||
}
|
||||
if (extname(name).toLowerCase() !== '.pdf') {
|
||||
return parseDocument(name, buffer, signal)
|
||||
}
|
||||
@@ -234,7 +239,7 @@ export class DocumentParsingService {
|
||||
? ['文档提取文本超过 5,000,000 字符,已截断']
|
||||
: [])
|
||||
]
|
||||
return buildPdfDocument(
|
||||
return buildPagedDocument(
|
||||
name,
|
||||
native,
|
||||
extracted.pageCount,
|
||||
@@ -246,7 +251,7 @@ export class DocumentParsingService {
|
||||
)
|
||||
}
|
||||
if (ocrPageNumbers.length === 0) {
|
||||
return buildPdfDocument(
|
||||
return buildPagedDocument(
|
||||
name,
|
||||
nativePdfSections(pages),
|
||||
extracted.pageCount,
|
||||
@@ -270,7 +275,7 @@ export class DocumentParsingService {
|
||||
purpose !== 'knowledge-index' &&
|
||||
native.some((section) => hasUsefulText(section.content))
|
||||
) {
|
||||
return buildPdfDocument(
|
||||
return buildPagedDocument(
|
||||
name,
|
||||
native,
|
||||
extracted.pageCount,
|
||||
@@ -308,7 +313,7 @@ export class DocumentParsingService {
|
||||
) {
|
||||
const detail =
|
||||
error instanceof Error ? error.message : '本地 OCR 识别失败'
|
||||
return buildPdfDocument(
|
||||
return buildPagedDocument(
|
||||
name,
|
||||
native,
|
||||
extracted.pageCount,
|
||||
@@ -367,7 +372,7 @@ export class DocumentParsingService {
|
||||
}]
|
||||
: []
|
||||
})
|
||||
return buildPdfDocument(
|
||||
return buildPagedDocument(
|
||||
name,
|
||||
merged,
|
||||
extracted.pageCount,
|
||||
@@ -380,6 +385,115 @@ export class DocumentParsingService {
|
||||
)
|
||||
}
|
||||
|
||||
private async parsePptx(
|
||||
name: string,
|
||||
buffer: Buffer,
|
||||
purpose: DocumentParsingPurpose,
|
||||
signal?: AbortSignal
|
||||
): Promise<ParsedDocument> {
|
||||
const settings = await this.settingsStore.get()
|
||||
const mode = effectiveOcrMode(settings, purpose)
|
||||
if (mode === 'disabled') {
|
||||
try {
|
||||
return await parseDocument(name, buffer, signal)
|
||||
} catch (error) {
|
||||
if (error instanceof DocumentTextUnavailableError) {
|
||||
throw new DocumentTextUnavailableError(
|
||||
'PPTX 没有可用文本,当前工作流未启用 OCR'
|
||||
)
|
||||
}
|
||||
throw error
|
||||
}
|
||||
}
|
||||
const pages = extractPptxPages(buffer)
|
||||
ensureNotAborted(signal)
|
||||
const native: ParsedSection[] = pages
|
||||
.filter((page) => page.content)
|
||||
.map((page) => ({
|
||||
locator: `幻灯片 ${page.pageNumber}`,
|
||||
content: page.content,
|
||||
method: 'native',
|
||||
pageNumber: page.pageNumber,
|
||||
blockKind: 'slide'
|
||||
}))
|
||||
const imagePages = pages.filter((page) => page.images.length > 0)
|
||||
if (imagePages.length > settings.maximumPages) {
|
||||
throw new Error(
|
||||
`PPTX 有 ${imagePages.length} 页需要 OCR,超过 ${settings.maximumPages} 页限制`
|
||||
)
|
||||
}
|
||||
const sections = [...native]
|
||||
const warnings: string[] = []
|
||||
let extractedCharacters = native.reduce(
|
||||
(total, section) => total + section.content.length, 0
|
||||
)
|
||||
if (imagePages.length > 0) {
|
||||
try {
|
||||
const status = await this.modelManager.getStatus(settings.localOcrModelId)
|
||||
if (!status.available || !status.verified) {
|
||||
throw new Error(status.detail)
|
||||
}
|
||||
ocrPages: for (const page of imagePages) {
|
||||
for (const [index, image] of page.images.entries()) {
|
||||
ensureNotAborted(signal)
|
||||
const locator = `幻灯片 ${page.pageNumber} · 图片 ${index + 1}`
|
||||
if (!image.mimeType) {
|
||||
throw new Error(`${locator} 的格式暂不支持 OCR:${extname(image.name)}`)
|
||||
}
|
||||
const request = {
|
||||
modelId: settings.localOcrModelId,
|
||||
fileName: image.name,
|
||||
mimeType: image.mimeType,
|
||||
data: Uint8Array.from(image.data).buffer,
|
||||
maximumPages: 1,
|
||||
pageTimeoutSeconds: settings.pageTimeoutSeconds
|
||||
}
|
||||
const result = await (signal
|
||||
? this.ocrBroker.recognize(request, signal)
|
||||
: this.ocrBroker.recognize(request))
|
||||
ensureNotAborted(signal)
|
||||
sections.push(...result.sections.map((section): ParsedSection => ({
|
||||
locator,
|
||||
content: section.content,
|
||||
confidence: section.confidence,
|
||||
method: 'ocr',
|
||||
pageNumber: page.pageNumber,
|
||||
blockKind: 'slide'
|
||||
})))
|
||||
for (const warning of result.warnings) {
|
||||
if (warnings.length < maximumDocumentParsingWarnings) {
|
||||
warnings.push(`${locator}:${warning}`.slice(0, 500))
|
||||
}
|
||||
}
|
||||
extractedCharacters += result.sections.reduce(
|
||||
(total, section) => total + section.content.length, 0
|
||||
)
|
||||
if (extractedCharacters >= maximumDocumentExtractedCharacters) {
|
||||
warnings.push('文档提取文本超过 5,000,000 字符,已截断')
|
||||
break ocrPages
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
ensureNotAborted(signal)
|
||||
if (
|
||||
mode !== 'auto' ||
|
||||
purpose === 'knowledge-index' ||
|
||||
!native.some((section) => hasUsefulText(section.content))
|
||||
) {
|
||||
throw error
|
||||
}
|
||||
return buildPagedDocument(name, native, pages.length, [
|
||||
`本地 OCR 不可用,已保留 PPTX 文本内容:${
|
||||
error instanceof Error ? error.message : '本地 OCR 识别失败'
|
||||
}`.slice(0, 500)
|
||||
], '.pptx')
|
||||
}
|
||||
}
|
||||
sections.sort((left, right) => (left.pageNumber ?? 0) - (right.pageNumber ?? 0))
|
||||
return buildPagedDocument(name, sections, pages.length, warnings, '.pptx')
|
||||
}
|
||||
|
||||
async diagnose(
|
||||
name: string,
|
||||
buffer: Buffer,
|
||||
@@ -387,9 +501,9 @@ export class DocumentParsingService {
|
||||
): Promise<DocumentParsingDiagnostic> {
|
||||
const startedAt = Date.now()
|
||||
const parsed = await this.parse(name, buffer, purpose)
|
||||
const ocrPageCount = parsed.sections.filter(
|
||||
const ocrPageCount = new Set(parsed.sections.filter(
|
||||
(section) => section.method === 'ocr'
|
||||
).length
|
||||
).map((section) => section.pageNumber ?? section.locator)).size
|
||||
const nativePageCount = parsed.sections.filter(
|
||||
(section) => section.method !== 'ocr'
|
||||
).length
|
||||
@@ -398,7 +512,7 @@ export class DocumentParsingService {
|
||||
sourceFormat:
|
||||
parsed.sourceFormat.replace(/^\./u, '').toUpperCase() || 'UNKNOWN',
|
||||
pageCount:
|
||||
parsed.sourceFormat === '.pdf'
|
||||
parsed.sourceFormat === '.pdf' || parsed.sourceFormat === '.pptx'
|
||||
? (parsed.pageCount ?? parsed.sections.length)
|
||||
: 0,
|
||||
ocrPageCount,
|
||||
|
||||
@@ -102,7 +102,7 @@ function decodeXmlEntities(value: string): string {
|
||||
)
|
||||
}
|
||||
|
||||
function extractXmlText(xml: string): string {
|
||||
export function extractXmlText(xml: string): string {
|
||||
return decodeXmlEntities(
|
||||
xml
|
||||
.replace(/<w:tab\b[^>]*\/>/g, '\t')
|
||||
@@ -239,19 +239,10 @@ function extractWorksheetText(
|
||||
.trim()
|
||||
}
|
||||
|
||||
function parseOfficeArchive(
|
||||
export function readOfficeArchive(
|
||||
buffer: Buffer,
|
||||
extension: string
|
||||
): ParsedSection[] {
|
||||
const patterns =
|
||||
extension === '.docx'
|
||||
? [/^word\/document\.xml$/]
|
||||
: extension === '.xlsx'
|
||||
? [
|
||||
/^xl\/sharedStrings\.xml$/,
|
||||
/^xl\/worksheets\/sheet\d+\.xml$/
|
||||
]
|
||||
: [/^ppt\/slides\/slide\d+\.xml$/]
|
||||
patterns: RegExp[]
|
||||
): Record<string, Uint8Array> {
|
||||
let archive: Record<string, Uint8Array>
|
||||
let entryCount = 0
|
||||
let selectedBytes = 0
|
||||
@@ -281,6 +272,24 @@ function parseOfficeArchive(
|
||||
} catch {
|
||||
throw new Error('Office 文档已损坏或不是有效的 Open XML 文件')
|
||||
}
|
||||
return archive
|
||||
}
|
||||
|
||||
function parseOfficeArchive(
|
||||
buffer: Buffer,
|
||||
extension: string
|
||||
): ParsedSection[] {
|
||||
const archive = readOfficeArchive(
|
||||
buffer,
|
||||
extension === '.docx'
|
||||
? [/^word\/document\.xml$/]
|
||||
: extension === '.xlsx'
|
||||
? [
|
||||
/^xl\/sharedStrings\.xml$/,
|
||||
/^xl\/worksheets\/sheet\d+\.xml$/
|
||||
]
|
||||
: [/^ppt\/slides\/slide\d+\.xml$/]
|
||||
)
|
||||
|
||||
if (extension === '.docx') {
|
||||
const documentXml = archive['word/document.xml']
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
import { strToU8, zipSync } from 'fflate'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { extractPptxPages } from './pptx-parser'
|
||||
import { createImagePptx } from '../../../tests/support/pptx-fixture'
|
||||
|
||||
describe('PPTX image extraction', () => {
|
||||
it('extracts referenced images with slide locators', () => {
|
||||
const pages = extractPptxPages(createImagePptx('Native text', 2))
|
||||
expect(pages.map((page) => page.pageNumber)).toEqual([1, 2])
|
||||
expect(pages[0]).toEqual({
|
||||
pageNumber: 1,
|
||||
content: 'Native text',
|
||||
images: [{
|
||||
name: 'image1.png',
|
||||
data: new Uint8Array([1, 2, 3]),
|
||||
mimeType: 'image/png'
|
||||
}]
|
||||
})
|
||||
})
|
||||
|
||||
it('follows presentation order and ignores external and unreferenced media', () => {
|
||||
const pages = extractPptxPages(Buffer.from(zipSync({
|
||||
'ppt/presentation.xml': strToU8('<p:presentation><p:sldIdLst><p:sldId r:id="second"/><p:sldId r:id="first"/></p:sldIdLst></p:presentation>'),
|
||||
'ppt/_rels/presentation.xml.rels': strToU8('<Relationships><Relationship Id="first" Target="slides/slide1.xml"/><Relationship Id="second" Target="/ppt/slides/slide2.xml"/></Relationships>'),
|
||||
'ppt/slides/slide1.xml': strToU8('<p:sld><a:t>First</a:t><a:blip r:embed="external"/></p:sld>'),
|
||||
'ppt/slides/_rels/slide1.xml.rels': strToU8('<Relationships><Relationship Id="external" Target="https://example.invalid/image.png" TargetMode="External"/></Relationships>'),
|
||||
'ppt/slides/slide2.xml': strToU8('<p:sld><a:t>Second</a:t></p:sld>'),
|
||||
'ppt/media/unused.png': new Uint8Array([1])
|
||||
})))
|
||||
expect(pages.map(({ pageNumber, content, images }) => ({
|
||||
pageNumber, content, images
|
||||
}))).toEqual([
|
||||
{ pageNumber: 1, content: 'Second', images: [] },
|
||||
{ pageNumber: 2, content: 'First', images: [] }
|
||||
])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,99 @@
|
||||
import { posix } from 'node:path'
|
||||
import { extractXmlText, readOfficeArchive } from './document-parser'
|
||||
|
||||
export type PptxPage = {
|
||||
pageNumber: number
|
||||
content: string
|
||||
images: Array<{
|
||||
name: string
|
||||
data: Uint8Array
|
||||
mimeType?: 'image/png' | 'image/jpeg' | 'image/webp'
|
||||
}>
|
||||
}
|
||||
|
||||
function attribute(tag: string, name: string): string | undefined {
|
||||
return new RegExp(`\\b${name}=["']([^"']*)["']`, 'u').exec(tag)?.[1]
|
||||
}
|
||||
|
||||
function relationships(xml: string, owner: string): Map<string, string> {
|
||||
const result = new Map<string, string>()
|
||||
for (const tag of xml.match(/<(?:\w+:)?Relationship\b[^>]*>/gu) ?? []) {
|
||||
const id = attribute(tag, 'Id')
|
||||
const target = attribute(tag, 'Target')
|
||||
if (!id || !target || attribute(tag, 'TargetMode') === 'External') {
|
||||
continue
|
||||
}
|
||||
result.set(
|
||||
id,
|
||||
posix.normalize(
|
||||
target.startsWith('/')
|
||||
? target.slice(1)
|
||||
: posix.join(posix.dirname(owner), target)
|
||||
)
|
||||
)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
export function extractPptxPages(buffer: Buffer): PptxPage[] {
|
||||
const archive = readOfficeArchive(buffer, [
|
||||
/^ppt\/presentation\.xml$/,
|
||||
/^ppt\/_rels\/presentation\.xml\.rels$/,
|
||||
/^ppt\/slides\/[^/]+\.xml$/,
|
||||
/^ppt\/slides\/_rels\/[^/]+\.xml\.rels$/,
|
||||
/^ppt\/media\/[^/]+$/
|
||||
])
|
||||
const text = (name: string): string =>
|
||||
archive[name] ? Buffer.from(archive[name]).toString('utf8') : ''
|
||||
const slideRelationships = relationships(
|
||||
text('ppt/_rels/presentation.xml.rels'),
|
||||
'ppt/presentation.xml'
|
||||
)
|
||||
const slideIds =
|
||||
text('ppt/presentation.xml').match(/<(?:\w+:)?sldId\b[^>]*>/gu) ?? []
|
||||
const paths = slideIds.length > 0
|
||||
? slideIds.map((tag) => {
|
||||
const path = slideRelationships.get(attribute(tag, 'r:id') ?? '')
|
||||
if (!path || !archive[path]) {
|
||||
throw new Error('PPTX 幻灯片关系无效')
|
||||
}
|
||||
return path
|
||||
})
|
||||
: Object.keys(archive)
|
||||
.filter((path) => /^ppt\/slides\/slide\d+\.xml$/u.test(path))
|
||||
.sort((left, right) => left.localeCompare(right, undefined, { numeric: true }))
|
||||
|
||||
return paths.map((path, index) => {
|
||||
const xml = text(path)
|
||||
const imageRelationships = relationships(
|
||||
text(posix.join(posix.dirname(path), '_rels', `${posix.basename(path)}.rels`)),
|
||||
path
|
||||
)
|
||||
const imagePaths = new Set(
|
||||
(xml.match(/<(?:\w+:)?blip\b[^>]*>/gu) ?? [])
|
||||
.map((tag) => imageRelationships.get(attribute(tag, 'r:embed') ?? ''))
|
||||
.filter((value): value is string => value !== undefined)
|
||||
)
|
||||
const images: PptxPage['images'] = []
|
||||
for (const imagePath of imagePaths) {
|
||||
const extension = posix.extname(imagePath).toLowerCase()
|
||||
const mimeType = extension === '.png'
|
||||
? 'image/png'
|
||||
: extension === '.jpg' || extension === '.jpeg'
|
||||
? 'image/jpeg'
|
||||
: extension === '.webp'
|
||||
? 'image/webp'
|
||||
: undefined
|
||||
const data = archive[imagePath]
|
||||
if (!data) {
|
||||
throw new Error(`幻灯片 ${index + 1} 的内嵌图片缺失`)
|
||||
}
|
||||
images.push({ name: posix.basename(imagePath), data, mimeType })
|
||||
}
|
||||
return {
|
||||
pageNumber: index + 1,
|
||||
content: extractXmlText(xml),
|
||||
images
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import type { DocumentOcrRequest } from '../../shared/document-parsing-contracts'
|
||||
|
||||
class TestWorker extends EventTarget {
|
||||
static instances: TestWorker[] = []
|
||||
terminate = vi.fn()
|
||||
postMessage = vi.fn((message: { type: string }) => {
|
||||
if (message.type === 'initialize') {
|
||||
this.dispatchEvent(new MessageEvent('message', { data: { type: 'ready' } }))
|
||||
}
|
||||
})
|
||||
|
||||
constructor() {
|
||||
super()
|
||||
TestWorker.instances.push(this)
|
||||
}
|
||||
|
||||
finish(requestId: string): void {
|
||||
this.dispatchEvent(new MessageEvent('message', {
|
||||
data: { type: 'result', result: { requestId, sections: [], pageCount: 1, warnings: [] } }
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
function request(): DocumentOcrRequest {
|
||||
return {
|
||||
requestId: crypto.randomUUID(),
|
||||
modelId: 'pp-ocrv6-tiny',
|
||||
fileName: 'scan.png',
|
||||
mimeType: 'image/png',
|
||||
data: new ArrayBuffer(1),
|
||||
maximumPages: 1,
|
||||
pageTimeoutSeconds: 300
|
||||
}
|
||||
}
|
||||
|
||||
let dispose: (() => void) | undefined
|
||||
|
||||
beforeEach(() => {
|
||||
vi.resetModules()
|
||||
vi.useFakeTimers()
|
||||
TestWorker.instances = []
|
||||
vi.stubGlobal('Worker', TestWorker)
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
dispose?.()
|
||||
dispose = undefined
|
||||
await vi.advanceTimersByTimeAsync(0)
|
||||
vi.useRealTimers()
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
async function setup() {
|
||||
let receive!: (request: DocumentOcrRequest) => void
|
||||
let cancel!: (requestId: string) => void
|
||||
const getOcrAssets = vi.fn(async (modelId: string) => ({
|
||||
modelId, detection: new ArrayBuffer(1), recognition: new ArrayBuffer(1), dictionary: new ArrayBuffer(1)
|
||||
}))
|
||||
const respondOcr = vi.fn(async () => undefined)
|
||||
const removeRequest = vi.fn()
|
||||
const removeCancel = vi.fn()
|
||||
vi.stubGlobal('goodbuddy', {
|
||||
documentParsing: {
|
||||
getOcrAssets, respondOcr,
|
||||
onOcrRequest: (listener: typeof receive) => { receive = listener; return removeRequest },
|
||||
onOcrCancel: (listener: typeof cancel) => { cancel = listener; return removeCancel }
|
||||
}
|
||||
})
|
||||
const { installDocumentOcrBridge } = await import('./document-ocr-bridge')
|
||||
dispose = installDocumentOcrBridge()
|
||||
const start = async () => {
|
||||
const input = request()
|
||||
receive(input)
|
||||
await vi.advanceTimersByTimeAsync(0)
|
||||
return input
|
||||
}
|
||||
const finish = async (input: DocumentOcrRequest) => {
|
||||
TestWorker.instances.at(-1)!.finish(input.requestId)
|
||||
await vi.advanceTimersByTimeAsync(0)
|
||||
}
|
||||
return { start, finish, receive, cancel, getOcrAssets, respondOcr, removeRequest, removeCancel }
|
||||
}
|
||||
|
||||
describe('document OCR worker idle release', () => {
|
||||
it('releases after 60 idle seconds and reloads the model for the next request', async () => {
|
||||
const api = await setup()
|
||||
const input = await api.start()
|
||||
const worker = TestWorker.instances[0]!
|
||||
await api.finish(input)
|
||||
await vi.advanceTimersByTimeAsync(59_999)
|
||||
expect(worker.terminate).not.toHaveBeenCalled()
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
expect(worker.terminate).toHaveBeenCalledOnce()
|
||||
const next = await api.start()
|
||||
expect(TestWorker.instances).toHaveLength(2)
|
||||
expect(api.getOcrAssets).toHaveBeenCalledTimes(2)
|
||||
await api.finish(next)
|
||||
})
|
||||
|
||||
it('cancels the idle timer for new work and reuses the loaded worker', async () => {
|
||||
const api = await setup()
|
||||
await api.finish(await api.start())
|
||||
await vi.advanceTimersByTimeAsync(59_000)
|
||||
const next = await api.start()
|
||||
const worker = TestWorker.instances[0]!
|
||||
await vi.advanceTimersByTimeAsync(61_000)
|
||||
expect(worker.terminate).not.toHaveBeenCalled()
|
||||
expect(api.getOcrAssets).toHaveBeenCalledOnce()
|
||||
await api.finish(next)
|
||||
await vi.advanceTimersByTimeAsync(60_000)
|
||||
expect(worker.terminate).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('does not release while a second request is queued or running', async () => {
|
||||
const api = await setup()
|
||||
const first = await api.start()
|
||||
const second = await api.start()
|
||||
const worker = TestWorker.instances[0]!
|
||||
await vi.advanceTimersByTimeAsync(61_000)
|
||||
expect(worker.terminate).not.toHaveBeenCalled()
|
||||
await api.finish(first)
|
||||
await vi.advanceTimersByTimeAsync(61_000)
|
||||
expect(worker.terminate).not.toHaveBeenCalled()
|
||||
await api.finish(second)
|
||||
await vi.advanceTimersByTimeAsync(60_000)
|
||||
expect(worker.terminate).toHaveBeenCalledOnce()
|
||||
expect(api.respondOcr).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it('clears the idle timer on disposal and removes subscriptions', async () => {
|
||||
const api = await setup()
|
||||
await api.finish(await api.start())
|
||||
const worker = TestWorker.instances[0]!
|
||||
dispose?.()
|
||||
dispose = undefined
|
||||
expect(api.removeRequest).toHaveBeenCalledOnce()
|
||||
expect(api.removeCancel).toHaveBeenCalledOnce()
|
||||
await vi.advanceTimersByTimeAsync(60_000)
|
||||
expect(worker.terminate).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('still terminates immediately on cancellation', async () => {
|
||||
const api = await setup()
|
||||
const input = await api.start()
|
||||
const worker = TestWorker.instances[0]!
|
||||
api.cancel(input.requestId)
|
||||
await vi.advanceTimersByTimeAsync(0)
|
||||
expect(worker.terminate).toHaveBeenCalledOnce()
|
||||
expect(api.respondOcr).toHaveBeenCalledWith({
|
||||
requestId: input.requestId, error: '本地 OCR 解析已取消'
|
||||
})
|
||||
await vi.advanceTimersByTimeAsync(60_000)
|
||||
expect(worker.terminate).toHaveBeenCalledOnce()
|
||||
})
|
||||
})
|
||||
@@ -26,6 +26,26 @@ const pending = new Map<string, PendingWorkerRequest>()
|
||||
const cancelledRequestIds = new Set<string>()
|
||||
let activeRequestId: string | undefined
|
||||
let requestQueue: Promise<void> = Promise.resolve()
|
||||
let queuedRequestCount = 0
|
||||
let idleTimer: number | undefined
|
||||
const workerIdleTimeoutMs = 60_000
|
||||
|
||||
function clearIdleTimer(): void {
|
||||
window.clearTimeout(idleTimer)
|
||||
idleTimer = undefined
|
||||
}
|
||||
|
||||
function scheduleIdleRelease(): void {
|
||||
clearIdleTimer()
|
||||
if (!worker || queuedRequestCount > 0 || activeRequestId || pending.size > 0) {
|
||||
return
|
||||
}
|
||||
idleTimer = window.setTimeout(() => {
|
||||
if (queuedRequestCount === 0 && !activeRequestId && pending.size === 0) {
|
||||
terminateWorker(new Error('本地 OCR 空闲资源已释放'))
|
||||
}
|
||||
}, workerIdleTimeoutMs)
|
||||
}
|
||||
|
||||
function safeError(error: unknown): string {
|
||||
return error instanceof Error
|
||||
@@ -34,6 +54,7 @@ function safeError(error: unknown): string {
|
||||
}
|
||||
|
||||
function terminateWorker(error: Error): void {
|
||||
clearIdleTimer()
|
||||
worker?.terminate()
|
||||
rejectWorkerReady?.(error)
|
||||
worker = undefined
|
||||
@@ -199,9 +220,15 @@ export function installDocumentOcrBridge(): () => void {
|
||||
return () => undefined
|
||||
}
|
||||
const removeRequestListener = api.onOcrRequest((request) => {
|
||||
clearIdleTimer()
|
||||
queuedRequestCount += 1
|
||||
requestQueue = requestQueue
|
||||
.then(() => handleRequest(request))
|
||||
.catch(() => undefined)
|
||||
.finally(() => {
|
||||
queuedRequestCount -= 1
|
||||
scheduleIdleRelease()
|
||||
})
|
||||
})
|
||||
const removeCancelListener = api.onOcrCancel((requestId) => {
|
||||
cancelledRequestIds.add(requestId)
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
import { strToU8, zipSync } from 'fflate'
|
||||
|
||||
export function createImagePptx(nativeText = '', pageCount = 1): Buffer {
|
||||
const files: Record<string, Uint8Array> = {
|
||||
'ppt/media/image1.png': new Uint8Array([1, 2, 3])
|
||||
}
|
||||
for (let page = 1; page <= pageCount; page += 1) {
|
||||
files[`ppt/slides/slide${page}.xml`] = strToU8(
|
||||
`<p:sld><a:p><a:t>${nativeText}</a:t></a:p><p:pic><a:blip r:embed="rId1"/></p:pic></p:sld>`
|
||||
)
|
||||
files[`ppt/slides/_rels/slide${page}.xml.rels`] = strToU8(
|
||||
'<Relationships><Relationship Id="rId1" Target="../media/image1.png"/></Relationships>'
|
||||
)
|
||||
}
|
||||
return Buffer.from(zipSync(files))
|
||||
}
|
||||
Reference in New Issue
Block a user