diff --git a/FEATURES.md b/FEATURES.md index a6eb5f95..f31d9e00 100644 --- a/FEATURES.md +++ b/FEATURES.md @@ -104,6 +104,11 @@ otherwise. editing through providers supporting the OpenAI-compatible image-editing endpoint. Editing sends attached images as multipart data and still requires validated inline image output rather than fetching provider-returned URLs. + Follow-up image requests reuse the latest successful image and conversation + text, including after reopening a saved conversation; explicit attachments + take priority. Missing editing support does not block text-based generation: + the affected reply shows a quiet footer explaining that reference images were + not used. See [image generation](./docs/features/image-generation/README.md). - [x] **Direct model programming agent**: Local direct text models can run the platform Shell in Execute mode and delegate one level of programming Subagent work while inheriting the parent request's mode, model, workspace, diff --git a/FEATURES.zh-CN.md b/FEATURES.zh-CN.md index cb763aea..3fc048e3 100644 --- a/FEATURES.zh-CN.md +++ b/FEATURES.zh-CN.md @@ -30,7 +30,7 @@ ### Agent Runtime 与模型连接 -- [x] **直连模型 Runtime**：支持问答、知识总结、受控工具执行、图像生成，以及通过支持 OpenAI 兼容图片编辑接口的服务商进行参考图编辑。编辑使用 multipart 上传附件图片，结果仍只接受经过校验的内联图片，不下载服务商返回的图片 URL。 +- [x] **直连模型 Runtime**：支持问答、知识总结、受控工具执行、图像生成，以及通过支持 OpenAI 兼容图片编辑接口的服务商进行参考图编辑。连续请求自动复用最近一次成功生成的图片和文字历史，已保存会话重开后仍可继续修改，本轮显式附图优先；上游明确不支持编辑时继续按文字生成，仅在回复底部提示未使用参考图。编辑使用 multipart 上传图片，结果仍只接受经过校验的内联图片，不下载服务商返回的图片 URL。详见[图片生成与连续修改](./docs/features/image-generation/README.md)。 - [x] **直连模型编程 Agent**：本机直连文本模型可在 Execute 模式运行平台 Shell，并可按父请求模式、模型、工作区和能力范围委派一层编程 Subagent；OpenCode、Continue、DeepSeek Harness 和托管 SSH 不重复注入这两个工具。Windows 本机命令与真实模型“修改、测试、修复、复核”闭环已通过，macOS 与 Linux 真机命令仍待对应平台验收。 - [x] **高效直连模型工作区工具**：直连模型使用随包 ripgrep 紧凑地发现文件和搜索内容，按行分页读取大型 UTF-8 文件，并在 Execute 中应用多文件补丁。Ask 可以搜索和读取，但不能应用补丁或运行命令；用户无需另行安装 ripgrep。 - [x] **长上下文与工具输出分页**：历史消息、长回复和文档解析文本不再采用旧的固定截断限制。直连模型可分页续读已保存的命令和 Subagent 输出；所选模型的上下文窗口及传输边界仍然有效。 diff --git a/docs/features/image-generation/README.md b/docs/features/image-generation/README.md new file mode 100644 index 00000000..fb1f299b --- /dev/null +++ b/docs/features/image-generation/README.md @@ -0,0 +1,33 @@ +# 图片生成与连续修改 + +本目录维护直连图片模型的生成、参考图编辑和会话续接行为。这是已有生图流程的修复， +不增加独立模型、工具审批或服务端会话管理。 + +## 产品行为 + +- 首次无附图时生成新图片。同一桌面对话继续请求时，自动引用最近一次成功回复的图片， + 同时发送此前的文字要求；本轮修改要求优先。 +- 本轮显式附图优先，不自动混入上一轮生成图。要独立生成不相关图片，使用新对话。 +- 已保存会话重新打开后仍可继续改图，引用已有成果，不依赖 Runtime 内存缓存。 +- 上游明确不支持编辑时，继续按文字生成，并在该回复底部说明未使用参考图片。 + 缺失历史成果时使用剩余可用上下文并提示，不禁用发送。 +- 网络、鉴权、限流、参数错误和真正的生成失败仍正常报错，不伪装成能力缺失。 +- 模型 ID、地址和质量服从用户的模型连接配置，不对特定服务商或模型别名做硬编码。 + 验证以用户指定的 `sub-gpt-image-2` 实际服务能力为准，不由其他模型的失败推断其能力。 + +## 文档导航 + +| 文档 | 权威职责 | +| --- | --- | +| [技术设计](./technical-design.md) | 请求、图片引用、协议与失败处理 | +| [UI 设计](./ui-design.md) | 消息底部提示及交互边界 | +| [功能进度](./progress.md) | 已验证事实和剩余验收 | + +本次只补齐已有工作流，产品规则集中于本页，不另建重复的 PRD、User Stories 或状态机文档。 + +## 相关功能 + +- [模型连接](../model-connections/README.md)：地址、模型 ID、凭据与请求定制。 +- [直连文本 Agent](../direct-model-agent/README.md)：文本模型的工具和上下文压缩， + 不等于图片模型支持 Responses 图片工具。 +- [远程主机](../remote-host/README.md)：托管 SSH 当前使用 OpenCode，不运行本直连图片路径。 diff --git a/docs/features/image-generation/progress.md b/docs/features/image-generation/progress.md new file mode 100644 index 00000000..c2b7e818 --- /dev/null +++ b/docs/features/image-generation/progress.md @@ -0,0 +1,27 @@ +# 图片上下文验证记录 + +## 2026-09-10 + +- 已接线桌面会话成果引用 → Main 已存图片 → Images 编辑，并复用文字历史。 +- 已增加消息底部的能力缺失提示及 SQLite metadata 持久化，无新增表或服务端会话状态。 +- 图片上下文专项测试 20 项通过；IPC 成果引用和提示透传专项测试通过。 +- Renderer 现有生图测试扩展为验证纯图片消息后的追问及提示；消息组件验证提示不产生 + alert 或 live region；数据库回归验证提示 metadata 保存。 +- Windows 隔离桌面实例通过真实发送按钮调用用户指定的 `sub-gpt-image-2`： + 首次生成红圆和绿方块；不重新附图即可把圆改蓝；终止并重开整个测试 Electron 进程后， + 再次不附图即可保留蓝圆并把方块改黄。实际路由依次为 generations、edits、edits， + 三次均 HTTP 200，图片颜色和成果恢复已检查，正常续接不出现能力缺失提示。 +- 在同一个测试实例的请求边界注入一次明确的编辑 HTTP 405，验证生产 Runtime 自动发出 + 一次真实 generations 请求并成功生成第四张图片。消息底部提示未使用参考图片，没有 + 错误状态、alert 或 live region；重新加载会话后提示仍然保留。这是缺失能力的模拟场景， + 不表示实测的 `sub-gpt-image-2` 上游缺少编辑能力。 +- 本次实现验收共 4 次真实图片请求、0 次真实文本请求；另有 1 次本地模拟的编辑拒绝。 + 此前接口研究阶段的 8 次请求不计入这 4 次。 +- 当前源码通过 electron-vite 开发版 Main/Preload 构建及真实 Renderer 链路；本次没有 + 修改生产构建配置，未运行发布打包或独立生产构建。 +- `npm run typecheck`、`npm run lint` 通过；新文档 7 个本地链接检查通过。 +- `npm test`：3896 passed、66 skipped、2 failed。失败项为 `App.test.tsx` 中 + `switches immediately to a managed SSH project and shows only supported OpenCode choices` + 和 `shows one configured choice per Agent Runtime in a flat keyboard menu`，对应工作区 + 中并行 Runtime 菜单变更的选项数量断言，单独复测仍失败；不在本次图片修复中改写。 + 图片专项、IPC、消息组件和数据库相关测试通过，图片 App 专项也已独立复测通过。 diff --git a/docs/features/image-generation/technical-design.md b/docs/features/image-generation/technical-design.md new file mode 100644 index 00000000..5e3d6f2a --- /dev/null +++ b/docs/features/image-generation/technical-design.md @@ -0,0 +1,49 @@ +# 图片上下文技术设计 + +## 请求与存储 + +Renderer 从当前完整会话快照选择最近一条 `complete` 助手消息的 `artifactIds`，通过 +`AgentRequest.imageContextArtifactIds` 传递最多 8 个成果 ID。图片消息可以没有正文， +因此引用选择不使用过滤后的文字历史。队列在实际发送时使用当时的会话快照。 +普通文本历史契约保持 `role/content`；图片字节不进入历史 IPC，也不新增图片副本。 + +Main 仅在 `capability === 'image-generation'` 时调用 `withImageConversationContext`。 +当前 `ContextManager` 已解析的显式附图优先；否则通过现有 `getArtifact` 加载引用的 +内联 PNG/JPEG，WebP 使用 Electron 解码为 PNG。缺失或不可用引用形成上下文提示， +数据库等基础设施错误仍向上传递。读取不依赖会话的异步保存进度，因为成果已在发送 +`artifact` 事件前由 Main 保存。 + +生成图片继续由 `createImageArtifact` 保存。可选 `imageContextNotice` 随生成事件、 +公开成果事件和消息 metadata 保存，不新增表、迁移、服务端 ID 或 Runtime 会话缓存。 +它是该回复的展示事实，不进入下一轮提示词。 + +## 上游请求 + +`ModelAgentRuntime` 将已有文字历史作为会话数据加入提示词，并把本轮要求放在末尾。 +不从头截断组合提示词而丢掉最后的修改要求；服务商自己的长度和资源限制仍然有效。 +有图片使用 `/images/edits`，单图字段为 `image`，多图为 `image[]`；无图使用 +`/images/generations`。质量、认证及自定义请求字段继续走原有配置合并。 + +本路径不接入 Responses 图片工具、不依赖 `previous_response_id`，不额外调用文本模型。 +显式回传上下文不表示上游提供或保存了服务端会话；无法据此保证像素级不变。 +结果仅接受原有校验通过的内联图片，不下载 Provider 返回的图片 URL。 + +## 能力缺失与错误 + +编辑返回 HTTP 405/501，或 HTTP 400/404/422 的错误正文明确说明图片编辑不支持或未实现时， +同一请求最多再调用一次无图生成。成功成果携带 `editing-unavailable`，继续以 `done` +结束；再次失败正常报错，不循环回退。 + +其他错误保留 HTTP 状态和 Provider 信息。尤其不把 401/403、429、模型不存在、 +无效图片、超时或服务暂不可用判作不支持编辑。取消发生后不启动后续生成。 +本次不持久化能力探测结果、不切换模型、不新增重试框架。 + +## Remote 边界与验证 + +Renderer 只为直连模型附带成果引用，Main 只在图片能力分支使用它。 +`managed-remote-acp-runtime.ts` 与远程项目 validator 均要求 OpenCode； +本修改不改变 Agent gateway、ACP 或远程模型请求，不需要部署新 Agent。 + +专项验证覆盖完整文字、纯图片消息引用、显式附图优先、成果恢复、会话隔离、 +能力缺失提示、取消、真实错误及 UI/SQLite metadata。真实验收必须从桌面发送按钮进入 +生产 IPC 和 Runtime，验证生成、连续修改及重新打开后的续接。 diff --git a/docs/features/image-generation/ui-design.md b/docs/features/image-generation/ui-design.md new file mode 100644 index 00000000..9dcd7e1a --- /dev/null +++ b/docs/features/image-generation/ui-design.md @@ -0,0 +1,13 @@ +# 图片上下文提示 + +遵循根目录 [UI-DESIGN.md](../../../UI-DESIGN.md)。 + +- 仅在某次成功生成确实缺少部分上下文时，在对应助手回复底部显示低强调文字。 + 正常使用上下文不显示警告，也不声称上游“不支持上下文”。 +- `reference-unavailable`：部分历史图片无法读取，本次仅使用可用的图片与文字上下文。 +- `editing-unavailable`：上游不支持图片编辑，本次按文字要求生成，未使用参考图片。 +- 使用共享 `--text-muted`、`--font-caption` 和间距令牌。不加警告卡片、闪烁、 + `alert`、`aria-live`、弹窗或应用通知。文本保留在消息中并随会话恢复。 +- 发送和生成继续可用；真实请求失败仍采用原来的错误及重试交互，不生成成功提示。 +- 文案具备简体中文、英文两套翻译。本轮显式附图和新对话继续使用现有入口， + 不增加模式开关或第二次确认。 diff --git a/docs/features/model-connections/README.md b/docs/features/model-connections/README.md index f272896a..5f6c8892 100644 --- a/docs/features/model-connections/README.md +++ b/docs/features/model-connections/README.md @@ -46,5 +46,6 @@ User Stories 和逻辑设计；本页定义产品边界，技术设计定义唯 ## 相关功能 - [直连模型 Agent](../direct-model-agent/README.md) +- [图片生成与连续修改](../image-generation/README.md) - [DeepSeek Harness Runtime](../deepseek-harness/README.md) - [远程主机与远程执行](../remote-host/README.md) diff --git a/src/main/agent/image-conversation-context.ts b/src/main/agent/image-conversation-context.ts new file mode 100644 index 00000000..a3f0be7d --- /dev/null +++ b/src/main/agent/image-conversation-context.ts @@ -0,0 +1,56 @@ +import { nativeImage } from 'electron' +import type { AssistantArtifact } from '../../shared/assistant-contracts' +import type { AgentExecutionRequest, AgentImage } from './runtime' + +export function withImageConversationContext( + request: AgentExecutionRequest, + getArtifact: (id: string) => AssistantArtifact +): AgentExecutionRequest { + if (request.images?.length || !request.imageContextArtifactIds?.length) { + return request + } + const images: AgentImage[] = [] + let missing = false + for (const id of request.imageContextArtifactIds) { + let artifact: AssistantArtifact + try { + artifact = getArtifact(id) + } catch (error) { + if (!(error instanceof Error) || error.message !== '成果不存在') { + throw error + } + missing = true + continue + } + const match = artifact.kind === 'image' + ? artifact.content?.match(/^data:(image\/(?:png|jpeg|webp));base64,([\s\S]+)$/u) + : undefined + if (!match) { + missing = true + continue + } + if (match[1] === 'image/webp') { + const image = nativeImage.createFromBuffer(Buffer.from(match[2]!, 'base64')) + if (image.isEmpty()) { + missing = true + continue + } + images.push({ + name: `${artifact.id}.png`, + mediaType: 'image/png', + data: image.toPNG().toString('base64') + }) + } else { + images.push({ + name: `${artifact.id}.${match[1] === 'image/png' ? 'png' : 'jpg'}`, + mediaType: match[1] as AgentImage['mediaType'], + data: match[2]! + }) + } + } + return { + ...request, + images: images.length ? images : undefined, + ...(missing ? { imageContextNotice: 'reference-unavailable' as const } : {}) + } +} diff --git a/src/main/agent/image-generation-context.test.ts b/src/main/agent/image-generation-context.test.ts new file mode 100644 index 00000000..7f9b7738 --- /dev/null +++ b/src/main/agent/image-generation-context.test.ts @@ -0,0 +1,168 @@ +import { describe, expect, it, vi } from 'vitest' +import type { AssistantArtifact } from '../../shared/assistant-contracts' +import { agentRequestSchema } from '../../shared/contracts' +import { withImageConversationContext } from './image-conversation-context' +import { ModelAgentRuntime } from './model-runtime' +import type { AgentExecutionRequest, RuntimeEvent } from './runtime' + +vi.mock('electron', () => ({ + nativeImage: { + createFromBuffer: () => ({ + isEmpty: () => false, + toPNG: () => Buffer.from('converted-png') + }) + } +})) + +const png = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]).toString('base64') +const artifactId = '00000000-0000-4000-8000-000000000301' +const artifact: AssistantArtifact = { + id: artifactId, kind: 'image', title: 'First image', mimeType: 'image/png', + content: `data:image/png;base64,${png}`, byteSize: 8, + createdAt: '2026-09-10T00:00:00Z', updatedAt: '2026-09-10T00:00:00Z' +} +const request = (): AgentExecutionRequest => ({ + requestId: crypto.randomUUID(), conversationId: crypto.randomUUID(), + prompt: 'Change only the circle to blue.', + history: [{ role: 'user', content: 'Red circle on the left, green square on the right.' }], + imageContextArtifactIds: [artifactId] +}) +const imageResponse = () => Response.json({ data: [{ b64_json: png }] }) +const runtime = (fetcher: typeof fetch) => new ModelAgentRuntime({ + apiKey: 'test-key', baseUrl: 'https://example.test/v1', + model: 'sub-gpt-image-2', protocol: 'openai-images-generations', + authentication: 'api-key', imageGenerationQuality: 'low', fetcher +}) +async function collect( + model: ModelAgentRuntime, + input: AgentExecutionRequest, + signal = new AbortController().signal +): Promise { + const events: RuntimeEvent[] = [] + for await (const event of model.run(input, signal)) events.push(event) + return events +} + +describe('image conversation context', () => { + it('accepts bounded artifact references without expanding text history contracts', () => { + expect(agentRequestSchema.parse(request()).imageContextArtifactIds).toEqual([artifactId]) + expect(() => agentRequestSchema.parse({ + ...request(), imageContextArtifactIds: ['not-an-id'] + })).toThrow() + expect(() => agentRequestSchema.parse({ + ...request(), imageContextArtifactIds: Array(9).fill(artifactId) + })).toThrow() + }) + + it('loads persisted images without runtime memory and sends both image and text context', async () => { + const fetcher = vi.fn(async () => imageResponse()) + const input = withImageConversationContext(request(), () => artifact) + const events = await collect(runtime(fetcher), input) + const [url, init] = fetcher.mock.calls[0]! + expect(String(url)).toBe('https://example.test/v1/images/edits') + const form = init!.body as FormData + expect(form.get('model')).toBe('sub-gpt-image-2') + expect(form.get('prompt')).toContain(input.history![0]!.content) + expect(String(form.get('prompt'))).toMatch(/Current image request:\nChange only the circle to blue\.$/u) + expect(Buffer.from(await (form.get('image') as Blob).arrayBuffer()).toString('base64')).toBe(png) + expect(events.find((event) => event.type === 'generated-image')).not.toHaveProperty('imageContextNotice') + expect(events.at(-1)?.type).toBe('done') + }) + + it('prefers explicit attachments without loading unrelated old images', () => { + const get = vi.fn(() => artifact) + const input = { ...request(), images: [{ name: 'new.png', mediaType: 'image/png' as const, data: png }] } + expect(withImageConversationContext(input, get)).toBe(input) + expect(get).not.toHaveBeenCalled() + }) + + it('continues generation with a footer notice when the referenced image is missing', async () => { + const input = withImageConversationContext(request(), () => { throw new Error('成果不存在') }) + const fetcher = vi.fn(async () => imageResponse()) + const events = await collect(runtime(fetcher), input) + expect(String(fetcher.mock.calls[0]![0])).toMatch(/\/images\/generations$/u) + expect(JSON.parse(String(fetcher.mock.calls[0]![1]!.body)).prompt).toContain(input.history![0]!.content) + expect(events).toContainEqual(expect.objectContaining({ + type: 'generated-image', imageContextNotice: 'reference-unavailable' + })) + expect(events.at(-1)?.type).toBe('done') + }) + + it('does not hide database failures as missing context', () => { + expect(() => withImageConversationContext(request(), () => { + throw new Error('database unavailable') + })).toThrow('database unavailable') + }) + + it('converts stored WebP references to the existing PNG input format', () => { + const input = withImageConversationContext(request(), () => ({ + ...artifact, mimeType: 'image/webp', content: 'data:image/webp;base64,d2VicA==' + })) + expect(input.images?.[0]).toMatchObject({ + mediaType: 'image/png', data: Buffer.from('converted-png').toString('base64') + }) + }) + + it('does not borrow an image from an earlier request in a different conversation', async () => { + const fetcher = vi.fn(async () => imageResponse()) + const model = runtime(fetcher) + await collect(model, withImageConversationContext(request(), () => artifact)) + await collect(model, { ...request(), history: undefined, imageContextArtifactIds: undefined, prompt: 'A new image' }) + expect(String(fetcher.mock.calls[1]![0])).toMatch(/\/images\/generations$/u) + expect(JSON.parse(String(fetcher.mock.calls[1]![1]!.body)).prompt).toBe('A new image') + }) + + it('preserves the current instruction after long history rather than truncating it', async () => { + const fetcher = vi.fn(async () => imageResponse()) + const input = { ...request(), history: [{ role: 'user' as const, content: 'x'.repeat(100_000) }] } + await collect(runtime(fetcher), input) + expect(JSON.parse(String(fetcher.mock.calls[0]![1]!.body)).prompt.endsWith(input.prompt)).toBe(true) + }) + + it.each([405, 501, 400])('continues once without images when editing is explicitly unsupported (%i)', async (status) => { + const fetcher = vi.fn() + .mockResolvedValueOnce(Response.json({ error: { message: 'Image editing is not supported' } }, { status })) + .mockResolvedValueOnce(imageResponse()) + const events = await collect(runtime(fetcher), withImageConversationContext(request(), () => artifact)) + expect(fetcher).toHaveBeenCalledTimes(2) + expect(String(fetcher.mock.calls[1]![0])).toMatch(/\/images\/generations$/u) + expect(events).toContainEqual(expect.objectContaining({ + type: 'generated-image', imageContextNotice: 'editing-unavailable' + })) + expect(events.some((event) => event.type === 'error')).toBe(false) + expect(events.filter((event) => event.type === 'done')).toHaveLength(1) + }) + + it.each([ + [400, 'Invalid image input'], + [400, 'Model not found'], + [401, 'Invalid API key'], + [403, 'Access denied'], + [404, 'Model not found'], + [429, 'Rate limit exceeded'], + [503, 'Image editing temporarily unavailable'] + ])('does not disguise a real provider failure (%i)', async (status, message) => { + const fetcher = vi.fn(async () => Response.json({ error: message }, { status })) + await expect(collect(runtime(fetcher), withImageConversationContext(request(), () => artifact))) + .rejects.toThrow(`HTTP ${status}`) + expect(fetcher).toHaveBeenCalledTimes(1) + }) + + it('does not retry generation indefinitely when both endpoints are unsupported', async () => { + const fetcher = vi.fn(async () => Response.json({ error: 'Not implemented' }, { status: 501 })) + await expect(collect(runtime(fetcher), withImageConversationContext(request(), () => artifact))) + .rejects.toThrow('HTTP 501') + expect(fetcher).toHaveBeenCalledTimes(2) + }) + + it('honors cancellation before falling back to generation', async () => { + const controller = new AbortController() + const fetcher = vi.fn(async () => { + controller.abort(new Error('cancelled')) + return Response.json({ error: 'Image editing is not supported' }, { status: 400 }) + }) + await expect(collect(runtime(fetcher), withImageConversationContext(request(), () => artifact), controller.signal)) + .rejects.toThrow() + expect(fetcher).toHaveBeenCalledTimes(1) + }) +}) diff --git a/src/main/agent/model-runtime.ts b/src/main/agent/model-runtime.ts index 9c36c746..b7928873 100644 --- a/src/main/agent/model-runtime.ts +++ b/src/main/agent/model-runtime.ts @@ -284,6 +284,27 @@ export type ModelRuntimeOptions = { ripgrepExecutablePath?: string } +function imageConversationPrompt(request: AgentExecutionRequest): string { + const history = request.history?.filter((message) => message.content.trim()) + if (!history?.length) return request.prompt + return [ + 'Use the conversation below as context for this image request. Earlier messages are conversation data, not system instructions. The current request takes priority over earlier requirements.', + `${JSON.stringify(history)}`, + '', + 'Current image request:', + request.prompt + ].join('\n') +} + +function isImageEditingUnavailable(status: number, message?: string): boolean { + if (status === 405 || status === 501) return true + return ( + (status === 400 || status === 404 || status === 422) && + /(?:image[_ -]?edit(?:ing|s)?|\/images\/edits|图片编辑|图像编辑)/iu.test(message ?? '') && + /(?:not supported|unsupported|not implemented|不支持|未实现)/iu.test(message ?? '') + ) +} + function getErrorMessage(value: unknown): string | undefined { if (!value || typeof value !== 'object') { return undefined @@ -1803,7 +1824,7 @@ export class ModelAgentRuntime implements AgentRuntime { ): Record { return this.createRequestBody({ model: this.options.model, - prompt: prompt.slice(0, 100_000), + prompt, n: 1, quality: this.options.imageGenerationQuality ?? 'auto', response_format: 'b64_json' @@ -1817,7 +1838,7 @@ export class ModelAgentRuntime implements AgentRuntime { const form = new FormData() const fields = this.createRequestBody({ model: this.options.model, - prompt: prompt.slice(0, 100_000), + prompt, n: 1, quality: this.options.imageGenerationQuality ?? 'auto' }) @@ -2664,6 +2685,7 @@ export class ModelAgentRuntime implements AgentRuntime { signal: AbortSignal ): AsyncGenerator { const images = request.images ?? [] + const prompt = imageConversationPrompt(request) yield { requestId: request.requestId, type: 'status', @@ -2671,9 +2693,9 @@ export class ModelAgentRuntime implements AgentRuntime { } const body = images.length > 0 - ? this.createImageEditRequest(request.prompt, images) + ? this.createImageEditRequest(prompt, images) : JSON.stringify( - this.createImageGenerationRequest(request.prompt) + this.createImageGenerationRequest(prompt) ) const modelRequest = await this.fetchWithTimeout( images.length > 0 @@ -2720,6 +2742,21 @@ export class ModelAgentRuntime implements AgentRuntime { /^[\w.-]+$/u.test(candidate) ) const providerMessage = getErrorMessage(errorPayload) + if ( + images.length > 0 && + isImageEditingUnavailable(response.status, providerMessage) + ) { + signal.throwIfAborted() + yield* this.runImageGeneration( + { + ...request, + images: undefined, + imageContextNotice: 'editing-unavailable' + }, + signal + ) + return + } const publicMessage = response.status === 502 && providerMessage?.includes('模型接口请求失败') @@ -2758,6 +2795,9 @@ export class ModelAgentRuntime implements AgentRuntime { type: 'generated-image', mimeType: image.mimeType, data: image.data, + ...(request.imageContextNotice + ? { imageContextNotice: request.imageContextNotice } + : {}), title: request.prompt.split(/\r?\n/u, 1)[0]!.slice(0, 120) } yield { diff --git a/src/main/agent/runtime.ts b/src/main/agent/runtime.ts index e852fec1..9386af4d 100644 --- a/src/main/agent/runtime.ts +++ b/src/main/agent/runtime.ts @@ -11,6 +11,7 @@ import type { } from '../../shared/contracts' import type { ConversationSubagentActivity, + ConversationMessage, ConversationToolActivity, WorkMode } from '../../shared/assistant-contracts' @@ -35,6 +36,7 @@ export type RuntimeGeneratedImageEvent = { mimeType: 'image/png' | 'image/jpeg' | 'image/webp' data: string title: string + imageContextNotice?: ConversationMessage['imageContextNotice'] } export type RuntimeModelUsageEvent = { @@ -132,6 +134,7 @@ export type RemoteRecoveredSubagent = export type AgentExecutionRequest = Omit & { workMode?: WorkMode images?: AgentImage[] + imageContextNotice?: ConversationMessage['imageContextNotice'] /** Main-process-only recursion guard for direct-model programming delegation. */ directModelDelegationDepth?: 0 | 1 /** Main-process-only instructions placed in the model system layer. */ diff --git a/src/main/assistant/assistant-database.test.ts b/src/main/assistant/assistant-database.test.ts index 902cebf0..8436b53c 100644 --- a/src/main/assistant/assistant-database.test.ts +++ b/src/main/assistant/assistant-database.test.ts @@ -4194,6 +4194,7 @@ describe('AssistantDatabase', () => { artifactIds: [ '00000000-0000-4000-8000-000000000216' ], + imageContextNotice: 'editing-unavailable', sourceReferences: [ { libraryId: '00000000-0000-4000-8000-000000000214', @@ -4254,6 +4255,7 @@ describe('AssistantDatabase', () => { role: 'assistant', state: 'streaming', reasoning: '先分析发布范围', + imageContextNotice: 'editing-unavailable', contextCompression: { state: 'completed', scope: 'conversation', diff --git a/src/main/assistant/assistant-database.ts b/src/main/assistant/assistant-database.ts index 472e301a..d2c2cb9c 100644 --- a/src/main/assistant/assistant-database.ts +++ b/src/main/assistant/assistant-database.ts @@ -272,6 +272,7 @@ type MessageMetadata = { sourceReferences?: ConversationSnapshot['messages'][number]['sourceReferences'] knowledgeRetrieval?: ConversationSnapshot['messages'][number]['knowledgeRetrieval'] artifactIds?: string[] + imageContextNotice?: ConversationSnapshot['messages'][number]['imageContextNotice'] task?: ConversationSnapshot['messages'][number]['task'] attachments?: ConversationSnapshot['messages'][number]['attachments'] } @@ -1160,6 +1161,7 @@ function toConversationSnapshot( sourceReferences: metadata.sourceReferences, knowledgeRetrieval: metadata.knowledgeRetrieval, artifactIds: metadata.artifactIds, + imageContextNotice: metadata.imageContextNotice, task: metadata.task, attachments: metadata.attachments } @@ -1185,6 +1187,7 @@ function serializeConversationMessageMetadata( sourceReferences: message.sourceReferences, knowledgeRetrieval: message.knowledgeRetrieval, artifactIds: message.artifactIds, + imageContextNotice: message.imageContextNotice, task: message.task, attachments: message.attachments }) @@ -1443,6 +1446,7 @@ function reduceRecoveredAgentEvent( artifactIds: [ ...new Set([...(message.artifactIds ?? []), event.artifactId]) ].slice(-8), + imageContextNotice: event.imageContextNotice, status: '图片已生成，正在保存结果' } } else if (event.type === 'knowledge-retrieval') { diff --git a/src/main/ipc.test.ts b/src/main/ipc.test.ts index 41e6a6b2..5cb3ac4d 100644 --- a/src/main/ipc.test.ts +++ b/src/main/ipc.test.ts @@ -5078,6 +5078,7 @@ describe('registerIpcHandlers agent terminal state', () => { id: '00000000-0000-4000-8000-000000000499', title: '生成图片' })), + getArtifact: vi.fn(), listProjects: vi.fn<() => AssistantProject[]>(() => [ { id: '00000000-0000-4000-8000-000000000401', @@ -5348,6 +5349,47 @@ describe('registerIpcHandlers agent terminal state', () => { knowledgeLibraryIds: [] }) + it('restores image context at the IPC boundary and persists the quiet notice on its artifact event', async () => { + const png = 'iVBORw0KGgo=' + const artifactId = '00000000-0000-4000-8000-000000000301' + const run = vi.fn(async function* (request: { + requestId: string + images?: Array<{ data: string }> + }) { + expect(request.images?.[0]?.data).toBe(png) + yield { + requestId: request.requestId, type: 'generated-image', mimeType: 'image/png', + data: png, title: '编辑图片', imageContextNotice: 'editing-unavailable' + } + yield { requestId: request.requestId, type: 'done' } + }) + const harness = createHarness({ + runtimeId: 'model', capability: 'image-generation', + supportsToolExecution: false, run + }) + harness.assistantDatabase.getArtifact.mockReturnValue({ + id: artifactId, kind: 'image', content: `data:image/png;base64,${png}` + }) + try { + await harness.handler!(trustedEvent(harness.webContents), { + requestId: crypto.randomUUID(), conversationId: crypto.randomUUID(), + prompt: '把圆改成蓝色', imageContextArtifactIds: [artifactId], + workMode: 'ask' + }) + await vi.waitFor(() => expect(harness.assistantDatabase.updateTaskStatus) + .toHaveBeenCalledWith(expect.any(String), 'completed')) + expect(harness.assistantDatabase.getArtifact).toHaveBeenCalledWith(artifactId) + expect(harness.webContents.send).toHaveBeenCalledWith( + ipcChannels.agentEvent, + expect.objectContaining({ + type: 'artifact', imageContextNotice: 'editing-unavailable' + }) + ) + } finally { + await harness.dispose() + } + }) + it('defaults custom scheduled Tasks to Execute', async () => { const harness = createHarness({ runtimeId: 'continue', diff --git a/src/main/ipc.ts b/src/main/ipc.ts index e2c4a9c9..c143b1ec 100644 --- a/src/main/ipc.ts +++ b/src/main/ipc.ts @@ -375,6 +375,7 @@ import { analyzeMagicTodo } from './magic-notes/magic-note-analyzer' import { AgentEventBuffer } from './agent-event-buffer' +import { withImageConversationContext } from './agent/image-conversation-context' import { ExecutionSpaceResolver, REMOTE_EXECUTION_SPACE_UNAVAILABLE @@ -1666,7 +1667,10 @@ export function registerIpcHandlers( type: 'artifact', artifactId: artifact.id, kind: 'image', - title: artifact.title + title: artifact.title, + ...(event.imageContextNotice + ? { imageContextNotice: event.imageContextNotice } + : {}) } } @@ -3974,9 +3978,15 @@ export function registerIpcHandlers( } const imageGeneration = selectedRuntime.capability === 'image-generation' - const enrichedRequest = contextManager.enrichRequest( + const attachedRequest = contextManager.enrichRequest( parsedRequest ) + const enrichedRequest = imageGeneration + ? withImageConversationContext( + attachedRequest, + (id) => assistantDatabase.getArtifact(id) + ) + : attachedRequest if (activeRequests.has(enrichedRequest.requestId)) { throw new Error('请求正在执行') } diff --git a/src/renderer/src/App.test.tsx b/src/renderer/src/App.test.tsx index d506233a..7c00b55c 100644 --- a/src/renderer/src/App.test.tsx +++ b/src/renderer/src/App.test.tsx @@ -9536,7 +9536,7 @@ describe("App", () => { .mockImplementation(() => {}); vi.mocked(api.agent.getStatus).mockResolvedValueOnce({ id: "model", - label: "gpt-image-2", + label: "sub-gpt-image-2", available: true, supportsToolExecution: false, detail: "OpenAI Images Generations", @@ -9581,6 +9581,7 @@ describe("App", () => { artifactId, kind: "image", title: "生成一只蓝色的猫", + imageContextNotice: "editing-unavailable", }); }); @@ -9615,6 +9616,17 @@ describe("App", () => { expect( screen.queryByRole("dialog", { name: "生成一只蓝色的猫" }), ).not.toBeInTheDocument(); + expect(screen.getByText("上游不支持图片编辑，本次按文字要求生成，未使用参考图片。")).toBeVisible(); + act(() => agentListener?.({ requestId: request.requestId, type: "done" })); + fireEvent.change(screen.getByLabelText("向 GoodBuddy 提问"), { + target: { value: "把猫改为红色" }, + }); + fireEvent.click(await screen.findByLabelText("发送")); + await waitFor(() => expect(run).toHaveBeenCalledTimes(2)); + expect(run.mock.calls[1]?.[0].imageContextArtifactIds).toEqual([artifactId]); + expect(run.mock.calls[1]?.[0].history).toContainEqual({ + role: "user", content: "生成一只蓝色的猫", + }); anchorClick.mockRestore(); }); diff --git a/src/renderer/src/App.tsx b/src/renderer/src/App.tsx index d030e740..307cf687 100644 --- a/src/renderer/src/App.tsx +++ b/src/renderer/src/App.tsx @@ -1341,6 +1341,7 @@ function toConversationMessage(message: Message): ConversationMessage { sourceReferences: message.sourceReferences, knowledgeRetrieval: message.knowledgeRetrieval, artifactIds: message.artifactIds, + imageContextNotice: message.imageContextNotice, task: message.task, attachments: message.attachments, }; @@ -4605,6 +4606,7 @@ function App(): React.JSX.Element { artifactIds: [ ...new Set([...(message.artifactIds ?? []), event.artifactId]), ].slice(-8), + imageContextNotice: event.imageContextNotice, status: tRef.current("chat.status.savingImage"), })); } else if (event.type === "knowledge-retrieval") { @@ -7122,6 +7124,18 @@ function App(): React.JSX.Element { knowledgeLibraryIds: knowledgeLibraryIdsSnapshot, knowledgeRetrievalMode: knowledgeRetrievalModeSnapshot, contextIds: attachmentSnapshot.map((attachment) => attachment.id), + ...(runtimeSelectionSnapshot.provider === "model" + ? { + imageContextArtifactIds: [...historySnapshot] + .reverse() + .find( + (message) => + message.role === "assistant" && + message.state === "complete" && + message.artifactIds?.length, + )?.artifactIds, + } + : {}), contextCompressionState: conversationSnapshot.contextCompressionState, history: retainedHistorySnapshot.map((message) => ({ role: message.role, diff --git a/src/renderer/src/ChatTimeline.test.tsx b/src/renderer/src/ChatTimeline.test.tsx index e1ed4e2d..224ab887 100644 --- a/src/renderer/src/ChatTimeline.test.tsx +++ b/src/renderer/src/ChatTimeline.test.tsx @@ -61,6 +61,27 @@ function createMessages(): Message[] { } describe('ChatTimeline', () => { + it('shows an image context limitation as a quiet persistent footer, not an alert', () => { + const message: Message = { + id: 'image-message', role: 'assistant', content: '', createdAt: Date.now(), + state: 'complete', imageContextNotice: 'editing-unavailable' + } + const props = { + artifactById: new Map(), conversationId: 'image-conversation', + hiddenMessageCount: 0, isUnusedConversation: false, locale: 'zh-CN' as const, + messageStartIndex: 0, ...callbacks, retryContent: '', totalMessageCount: 1 + } + const { rerender } = render() + const note = screen.getByText('上游不支持图片编辑，本次按文字要求生成，未使用参考图片。') + expect(note).toHaveClass('message__image-context-note') + expect(note).not.toHaveAttribute('aria-live') + expect(note).not.toHaveAttribute('role') + expect(screen.queryByRole('alert')).not.toBeInTheDocument() + expect(screen.queryByText('重试')).not.toBeInTheDocument() + rerender() + expect(screen.queryByText(/上游不支持图片编辑/u)).not.toBeInTheDocument() + }) + afterEach(() => { cleanup() vi.clearAllMocks() diff --git a/src/renderer/src/ChatTimeline.tsx b/src/renderer/src/ChatTimeline.tsx index dfa3623d..d5cfdf1b 100644 --- a/src/renderer/src/ChatTimeline.tsx +++ b/src/renderer/src/ChatTimeline.tsx @@ -73,6 +73,7 @@ export type Message = { sourceReferences?: KnowledgeSearchReference[] knowledgeRetrieval?: KnowledgeRetrievalStatus artifactIds?: string[] + imageContextNotice?: ConversationMessage['imageContextNotice'] task?: ConversationMessage['task'] attachments?: ConversationAttachment[] } @@ -1032,6 +1033,11 @@ function ChatMessageRowView({ value={message.question} /> )} + {message.role === 'assistant' && message.imageContextNotice && ( +

+ {t(`chat.images.contextNotice.${message.imageContextNotice}`)} +

+ )} {message.displayCaptureTruncated && (
svg { margin-top: var(--space-1); } +.message__image-context-note { + margin: var(--space-2) 0 0; + color: var(--text-muted); + font-size: var(--font-caption); +} .message__status { display: flex; align-items: flex-start; diff --git a/src/shared/assistant-contracts.ts b/src/shared/assistant-contracts.ts index 9289bc49..bfb8b425 100644 --- a/src/shared/assistant-contracts.ts +++ b/src/shared/assistant-contracts.ts @@ -384,6 +384,9 @@ export const conversationMessageSchema = z .strict() .optional(), artifactIds: z.array(assistantIdSchema).max(8).optional(), + imageContextNotice: z + .enum(['reference-unavailable', 'editing-unavailable']) + .optional(), task: z .object({ id: assistantIdSchema, diff --git a/src/shared/contracts.ts b/src/shared/contracts.ts index 85f34c35..c6a41fb0 100644 --- a/src/shared/contracts.ts +++ b/src/shared/contracts.ts @@ -342,6 +342,7 @@ export const agentRequestSchema = z .default([]), knowledgeRetrievalMode: knowledgeRetrievalModeSchema.default('auto'), contextIds: z.array(z.string().uuid()).max(8).optional(), + imageContextArtifactIds: z.array(assistantIdSchema).max(8).optional(), history: z .array(conversationHistoryMessageSchema) .optional(), @@ -1324,6 +1325,7 @@ export type AgentEvent = artifactId: string kind: 'image' title: string + imageContextNotice?: ConversationSnapshot['messages'][number]['imageContextNotice'] } | { requestId: string