fix: retry transient supervisor model failures once

This commit is contained in:
mesalogo
2026-10-05 10:47:09 +08:00
parent b4df493be7
commit e50ca6d149
8 changed files with 346 additions and 50 deletions
@@ -10,6 +10,8 @@
回顾获得执行位置后,读取一次连接配置;同次执行中的批次、输出重试、汇总、故事与经验阶段保持该模型。修改监督者选择、应用默认模型或连接配置,仅影响后续执行。继续暂停或失败的回顾、单独重试故事、生成或重试建议时重新读取当前选择;跨重启不保存模型或凭据快照。显式选择的连接被删除、改为图片协议或缺少所需 API Key 时,执行报错并提示修改设置,不自动换成默认模型。 回顾获得执行位置后,读取一次连接配置;同次执行中的批次、输出重试、汇总、故事与经验阶段保持该模型。修改监督者选择、应用默认模型或连接配置,仅影响后续执行。继续暂停或失败的回顾、单独重试故事、生成或重试建议时重新读取当前选择;跨重启不保存模型或凭据快照。显式选择的连接被删除、改为图片协议或缺少所需 API Key 时,执行报错并提示修改设置,不自动换成默认模型。
对应 FR-S6。监督模型遇到可识别的临时失败时,统一等待 500ms 后重试一次:HTTP 429、500/502/503/504,以及短暂网络故障或响应流中断。取消、模型超时、认证、配置、额度或容量错误直接停止;无法确认可恢复的错误也不重试。失败尝试的部分正文丢弃,已报告用量和已保存批次保留。叶子输出校验后的缩块重试、导航输出重试保持原规则;请求生命周期见[模型阶段超时与调度](./technical-design.md#模型阶段超时与调度)。
## 整体逻辑架构 ## 整体逻辑架构
报告、监督叶子与导航整理共用监督专用 FIFO 并发池,普通聊天不入池;监督服务外层 run 仍串行。报告在排队前冻结超时,获槽后领取,实际模型请求前校验并刷新租约;报告提交或无变化后先释放槽位,再执行监督。监督超时在 run 创建时保存,节点获槽并解析 Runtime 后计时。阶段超时、租约、传输时限、后台检查与活动轮询各自独立,取值及生效规则见[当前设置合同](./review-scheduling-design.md#当前设置合同)。 报告、监督叶子与导航整理共用监督专用 FIFO 并发池,普通聊天不入池;监督服务外层 run 仍串行。报告在排队前冻结超时,获槽后领取,实际模型请求前校验并刷新租约;报告提交或无变化后先释放槽位,再执行监督。监督超时在 run 创建时保存,节点获槽并解析 Runtime 后计时。阶段超时、租约、传输时限、后台检查与活动轮询各自独立,取值及生效规则见[当前设置合同](./review-scheduling-design.md#当前设置合同)。
@@ -4,6 +4,17 @@
当前记录以已验证生产行为为准。监督者尚未覆盖全部 user stories。 当前记录以已验证生产行为为准。监督者尚未覆盖全部 user stories。
## 2026-10-05 监督模型临时失败重试
对应 FR-S6。`runSupervisionModel` 已接入一次可取消的临时失败重试,覆盖监督者各模型阶段;错误分类、清理及每次尝试的超时规则见[技术设计](./technical-design.md#模型阶段超时与调度)。直连文本请求保留 HTTP status,并交出失败流中已报告的用量。OpenCode/Continue、直连传输重试和叶子/导航输出重试未改。监督者仍在桌面 Main 直连模型,不经过远程 Agent 或模型桥。
验证结果:
- `npx vitest run src/main/assistant/supervision-model-retry.test.ts src/main/assistant/supervision-production.test.ts src/main/assistant/supervision-review.test.ts src/main/assistant/supervision-model-pool.test.ts src/main/agent/model-runtime.test.ts`:5 个文件,177 项通过,1 项原有跳过。覆盖 429、可恢复 5xx、部分正文后 terminated、连续失败最多两次包装层调用、认证/配置/容量/超时不重试、等待中取消、流和会话清理、用量保留,以及已保存叶子在失败和继续后不重算。
- `npx tsc --noEmit -p tsconfig.node.json`、`npx tsc --noEmit -p tsconfig.agent.json`、4 个改动 TypeScript 文件的定向 ESLint 和 `git diff --check` 通过。
请求验证使用生产 `ModelAgentRuntime` 和注入的假 fetch/响应流,数据保存使用内存 SQLite;外部模型调用为 0。未运行全量测试、真实模型或远程 Host 验证,未提交或推送。
## 2026-10-05 后端阶段审查 A03/A04/A05 ## 2026-10-05 后端阶段审查 A03/A04/A05
对应 SL-5、SL-8、FR-S6、FR-S10。生产候选读取现在将批次项目和跨项目设置传到只读 Worker;最终候选读取返回后检查暂停/取消;故事归属和拆分在原写事务内拒绝已移除或合并的目标。存储、停止与重试规则见[技术设计](./technical-design.md#单次执行与取消)。没有 schema 变更,原始消息、历史结果及人工确认字段保留。 对应 SL-5、SL-8、FR-S6、FR-S10。生产候选读取现在将批次项目和跨项目设置传到只读 Worker;最终候选读取返回后检查暂停/取消;故事归属和拆分在原写事务内拒绝已移除或合并的目标。存储、停止与重试规则见[技术设计](./technical-design.md#单次执行与取消)。没有 schema 变更,原始消息、历史结果及人工确认字段保留。
@@ -156,7 +167,7 @@ FR-S13 已接通监督者“设置 → 模型”、应用设置持久化和 Main
- 叶子批次输出不可用(无效 JSON、结构不符、引用不存在的来源或实体)时,在同一段待处理正文上缩小到 1/2、再到 1/4(不少于 500 字符)重试,最多 2 次;缩小只是先处理前一部分,其余正文留在待处理里,不跳过、不截断。 - 叶子批次输出不可用(无效 JSON、结构不符、引用不存在的来源或实体)时,在同一段待处理正文上缩小到 1/2、再到 1/4(不少于 500 字符)重试,最多 2 次;缩小只是先处理前一部分,其余正文留在待处理里,不跳过、不截断。
- 导航合并输出不可用时,用相同输入再问 1 次。 - 导航合并输出不可用时,用相同输入再问 1 次。
- 提供方错误、超时、取消和响应超出容量不重试,仍按原来的方式失败并保留已保存批次。 - 当时提供方错误、超时、取消和响应超出容量不重试,失败保留已保存批次。2026-10-05 起临时提供方错误采用上文的一次重试;超时、取消和响应超出容量仍不重试。
改后结果(DeepSeek `deepseek-flash`): 改后结果(DeepSeek `deepseek-flash`):
@@ -12,6 +12,8 @@
Main 的手动 IPC 和心跳完成回调均使用 `createProductionSupervisorService`,该工厂连接 `SupervisorService`、`SupervisionReviewStore`、共享模型池及无工具 Runtime。生产路径不再调用旧的 20 会话 × 20 消息 collector,也不使用旧服务分支的 100 条 / 48,000 字符裁剪。旧的有界服务注入方式仍供原有测试和实验脚本使用;心跳报告本身继续使用原报告预算。 Main 的手动 IPC 和心跳完成回调均使用 `createProductionSupervisorService`,该工厂连接 `SupervisorService`、`SupervisionReviewStore`、共享模型池及无工具 Runtime。生产路径不再调用旧的 20 会话 × 20 消息 collector,也不使用旧服务分支的 100 条 / 48,000 字符裁剪。旧的有界服务注入方式仍供原有测试和实验脚本使用;心跳报告本身继续使用原报告预算。
2026-10-05 起,监督模型包装层对可识别的临时失败统一重试一次,包含 429、可恢复 5xx 和网络/响应流中断;取消、模型超时、配置及容量错误不重试。每次尝试独立计时,等待可取消,成功批次保留。具体错误与生命周期合同见[技术设计](./technical-design.md#模型阶段超时与调度);下文旧草案中“不增加节点网络重试”的描述已由此规则替代。
### 已接入的调度与存储 ### 已接入的调度与存储
schema 47 增加运行配置、来源清单、成功批次、导航节点四张表和一个实时来源视图。迁移只创建新结构,保留此前迁移、结果及用户数据。清单只保存来源身份、版本、长度及增量起点;正文随后按主键和有界 `substr` 读取,派发顺序仍由 `(project_id, conversation_id, sequence, source)` 决定。 schema 47 增加运行配置、来源清单、成功批次、导航节点四张表和一个实时来源视图。迁移只创建新结构,保留此前迁移、结果及用户数据。清单只保存来源身份、版本、长度及增量起点;正文随后按主键和有界 `substr` 读取,派发顺序仍由 `(project_id, conversation_id, sequence, source)` 决定。
@@ -50,7 +52,7 @@ schema 47 增加运行配置、来源清单、成功批次、导航节点四张
| `supervisionReview.executionSeconds` | 300 | 30..3600 | 旧设置兼容字段,执行时忽略,设置页不再展示;恢复旧 run 同样不按此值暂停 | | `supervisionReview.executionSeconds` | 300 | 30..3600 | 旧设置兼容字段,执行时忽略,设置页不再展示;恢复旧 run 同样不按此值暂停 |
| `supervisionReview.crossProject` | 关闭 | 开关 | 创建回顾时冻结;关闭时每批只以本项目已有实体为候选,也不生成跨项目故事;开启后候选来自全部项目,故事归属可建立跨项目故事 | | `supervisionReview.crossProject` | 关闭 | 开关 | 创建回顾时冻结;关闭时每批只以本项目已有实体为候选,也不生成跨项目故事;开启后候选来自全部项目,故事归属可建立跨项目故事 |
| `supervisionReview.storyThreadEvents` | 20 | 4..500 | 创建回顾时冻结;功能的事件数低于此值时,模型提出的子线索并入功能 | | `supervisionReview.storyThreadEvents` | 20 | 4..500 | 创建回顾时冻结;功能的事件数低于此值时,模型提出的子线索并入功能 |
| `supervisorOrganizeTimeoutSeconds` | 240 | 30..600 | 创建回顾时冻结,节点获槽并解析 Runtime 后计时 | | `supervisorOrganizeTimeoutSeconds` | 240 | 30..600 | 创建回顾时冻结,节点获槽并解析 Runtime 后,每次模型尝试独立计时 |
| `heartbeatReportTimeoutSeconds` | 240 | 30..600 | 维持报告发起时冻结及既有租约规则 | | `heartbeatReportTimeoutSeconds` | 240 | 30..600 | 维持报告发起时冻结及既有租约规则 |
| `supervisorModelConcurrency` | 1 | 1..4 | 回顾保存接纳并发;共享池上限仍随应用设置实时变化,实际执行取较小值 | | `supervisorModelConcurrency` | 1 | 1..4 | 回顾保存接纳并发;共享池上限仍随应用设置实时变化,实际执行取较小值 |
@@ -55,9 +55,13 @@ schema 43 在结果上增加 `graph_snapshot_json`,保存当次实体名称、
监督整理超时由 run 创建时冻结;共享模型池仍采用实时设置上限。`supervision:pause` 取消该 run 的排队和在途工作,`supervision:resume` 继续已保存批次;监督整理输出在流中按 `supervisionReview.responseKiB` 检查响应容量,默认 1024 KiB,可在设置页调整为 100 至 16384 KiB。该容量每次请求读取当前设置,继续旧运行也采用新值;超限保留成功批次并提示调高后继续。生成摘要及事实内容不设独立短字符限制,监督旧摘要读取保留全文,具体边界见[生产接线](./review-scheduling-design.md#0-生产接线与剩余边界)。以下报告领取和租约规则继续适用。 监督整理超时由 run 创建时冻结;共享模型池仍采用实时设置上限。`supervision:pause` 取消该 run 的排队和在途工作,`supervision:resume` 继续已保存批次;监督整理输出在流中按 `supervisionReview.responseKiB` 检查响应容量,默认 1024 KiB,可在设置页调整为 100 至 16384 KiB。该容量每次请求读取当前设置,继续旧运行也采用新值;超限保留成功批次并提示调高后继续。生成摘要及事实内容不设独立短字符限制,监督旧摘要读取保留全文,具体边界见[生产接线](./review-scheduling-design.md#0-生产接线与剩余边界)。以下报告领取和租约规则继续适用。
手动和自动监督均在桌面 Main 调用生产工厂,Runtime 解析请求不携带项目、Runtime 选择或执行空间,也不再注入 `workMode`。`resolveRequestRuntime` 返回桌面默认 Runtime,不进入 SSH 项目或 `selectedRuntimes.getRuntime` 分支;内部分析使用无工具构造,回顾 scope 只筛选证据。摘要校验、合并和持久化在 Main 完成,single-flight、取消及 UI 控制不改变 `gbagent`、远端 Runtime 启动或桌面到 Agent 协议。 手动和自动监督均在桌面 Main 调用生产工厂,由 `resolveSupervisorRuntime` 解析上文的直连模型,不进入 SSH 项目或 `selectedRuntimes.getRuntime` 分支;内部分析使用无工具构造,回顾 scope 只筛选证据。摘要校验、合并和持久化在 Main 完成,single-flight、取消及 UI 控制不改变 `gbagent`、远端 Runtime 启动或桌面到 Agent 协议。
`ApplicationSettingsStore` 持久化报告和整理模型超时,均为 30..600 整数秒,默认 240。报告排队前冻结,监督创建 run 时冻结;计时从获槽及 Runtime 解析后开始,覆盖模型执行及其内部重试。流结束校验 signal 和完成事件,finally 释放临时会话,心跳报告保持原有 100KB 上限,监督整理使用上文的可调响应容量。采集、排队、保存和清理不计入模型超时。 `ApplicationSettingsStore` 持久化报告和整理模型超时,均为 30..600 整数秒,默认 240。报告排队前冻结,监督创建 run 时冻结;计时从获槽及 Runtime 解析后开始,覆盖单次 Runtime 执行及其内部重试。监督包装层的第二次尝试重新按同一秒数计时,500ms 等待和临时会话清理不计入该时限。流结束校验 signal 和完成事件,finally 释放临时会话,心跳报告保持原有 100KB 上限,监督整理使用上文的可调响应容量。采集、排队、保存和清理不计入模型超时。
`runSupervisionModel` 为提取、导航、故事、经验和建议共用一次临时错误重试,错误范围见[逻辑规则](./logic-design.md#监督者模型选择)。每次尝试使用新的 request/conversation ID 和隐藏任务,正文缓冲单独创建;用量按各次任务保存。失败后先清理计时器并释放临时会话,再等待;等待继续持有监督池槽位,受父 signal 和池取消控制,结束或失败后释放槽位。`error` 事件的 `status: cancelled` 优先于错误文字;抛出的结构化错误检查 status、statusCode 和 cause,字符串只识别明确的 HTTP 状态或已知网络/流中断格式。
直连文本请求保留 HTTP 错误的 status,并在响应流失败前交出已收到的用量;没有提供方用量时不估算补记。重试决策只在监督包装层,直连传输原有重试、OpenCode 和 Continue 策略不变。429 使用同一 500ms 等待,不新增 Retry-After、UI 或配置项。
`supervisorModelConcurrency` 为 1 至 4 的整数,默认 1。报告和监督整理共用 Main 内 `SupervisionModelPool` FIFO 池,普通聊天不入池;降低上限不终止在途请求。报告槽位从领取前持有至报告完成或无变化提交后,在下游监督开始前释放,避免并发 1 时相互等待。监督服务外层运行仍串行;该池不是完整分块调度器,也不提供跨聊天与监督的提供商全局并发、RPM、TPM 或 Retry-After 控制。 `supervisorModelConcurrency` 为 1 至 4 的整数,默认 1。报告和监督整理共用 Main 内 `SupervisionModelPool` FIFO 池,普通聊天不入池;降低上限不终止在途请求。报告槽位从领取前持有至报告完成或无变化提交后,在下游监督开始前释放,避免并发 1 时相互等待。监督服务外层运行仍串行;该池不是完整分块调度器,也不提供跨聊天与监督的提供商全局并发、RPM、TPM 或 Retry-After 控制。
+4 -2
View File
@@ -3168,9 +3168,9 @@ export class ModelAgentRuntime implements AgentRuntime {
} catch { } catch {
detail = undefined detail = undefined
} }
throw new Error( throw Object.assign(new Error(
detail ?? `模型接口请求失败(HTTP ${response.status})` detail ?? `模型接口请求失败(HTTP ${response.status})`
) ), { status: response.status })
} }
let receivedStop = false let receivedStop = false
@@ -3208,6 +3208,8 @@ export class ModelAgentRuntime implements AgentRuntime {
} }
return { answer, usage } return { answer, usage }
} catch (error) { } catch (error) {
const usageEvent = createUsageEvent(request.requestId, anthropic ? 'anthropic' : 'openai', this.options.model, usage)
if (usageEvent) yield usageEvent
return normalizeRequestError(error, modelRequest.timedOut()) return normalizeRequestError(error, modelRequest.timedOut())
} finally { } finally {
modelRequest.clear() modelRequest.clear()
@@ -0,0 +1,231 @@
import { afterEach, expect, it, vi } from 'vitest'
import type { AgentRuntime } from '../agent/runtime'
import { ModelAgentRuntime, noModelTools } from '../agent/model-runtime'
import { AssistantDatabase } from './assistant-database'
import { SupervisionModelPool } from './supervision-model-pool'
import { createProductionSuggestionPhraser } from './supervision-production'
const cleanups: Array<() => void> = []
afterEach(() => {
for (const cleanup of cleanups.splice(0)) cleanup()
vi.useRealTimers()
})
function fixture(run: AgentRuntime['run'], timeoutSeconds = 30) {
vi.useFakeTimers()
const db = new AssistantDatabase(':memory:')
db.initialize(process.cwd())
const pool = new SupervisionModelPool()
const release = vi.fn<NonNullable<AgentRuntime['releaseConversation']>>(async () => {})
const dispose = vi.fn(async () => {})
const runtime = { run: vi.fn(run), releaseConversation: release, dispose } as unknown as AgentRuntime
const statuses = vi.spyOn(db, 'updateTaskStatus')
const phrase = createProductionSuggestionPhraser(db, async () => ({ supervisorModelConcurrency: 1,
supervisorOrganizeTimeoutSeconds: timeoutSeconds }), async () => runtime, pool)
cleanups.push(() => { pool.dispose(); db.close() })
return { db, pool, runtime, release, dispose, statuses,
start: () => phrase({ systemInstruction: 'Suggest', outputContract: '{}', candidates: [] }) }
}
it.each([
Object.assign(new Error('Slow down'), { status: 429 }),
Object.assign(new Error('Gateway unavailable'), { statusCode: 502 }),
new TypeError('fetch failed', { cause: Object.assign(new Error('socket closed'), { code: 'ECONNRESET' }) }),
new TypeError('terminated'),
'模型接口请求失败(HTTP 503)',
'Gateway Timeout (HTTP 504)',
'TypeError: terminated',
'模型接口流式响应意外中断',
'Rate limit exceeded'
])('retries a transient failure once with fresh text, usage identities and cleanup: %s', async failure => {
let calls = 0
const finalized = vi.fn()
const f = fixture(async function* (input, signal) {
calls++
try {
yield { type: 'model-usage', requestId: input.requestId, callId: 'provider-call', runtime: 'model', provider: 'openai',
model: 'fake', inputTokens: 10, outputTokens: 2, cacheReadTokens: 0, cacheWriteTokens: 0 }
if (calls === 1) {
yield { type: 'text', requestId: input.requestId, delta: 'discard partial {' }
if (typeof failure === 'string') yield { type: 'error', status: 'failed', requestId: input.requestId, message: failure }
else throw failure
} else {
expect(f.release).toHaveBeenCalledTimes(1)
expect(signal.aborted).toBe(false)
yield { type: 'text', requestId: input.requestId, delta: 'clean result' }
yield { type: 'done', requestId: input.requestId }
}
} finally { finalized() }
})
const result = f.start()
await vi.advanceTimersByTimeAsync(499)
expect(calls).toBe(1)
expect(f.release).toHaveBeenCalledTimes(1)
await vi.advanceTimersByTimeAsync(1)
await expect(result).resolves.toBe('clean result')
expect(calls).toBe(2)
expect(finalized).toHaveBeenCalledTimes(2)
expect(f.release).toHaveBeenCalledTimes(2)
expect(new Set(f.release.mock.calls.map(args => args[0])).size).toBe(2)
expect(f.dispose).toHaveBeenCalledOnce()
expect(f.statuses.mock.calls.map(args => args[1])).toEqual(['failed', 'completed'])
expect(f.db.getTokenUsageSummary().systemTotals).toMatchObject({ callCount: 2, input: 20, output: 4 })
expect(f.db.listTasks()).toEqual([])
await expect(f.pool.run(async () => 'released')).resolves.toBe('released')
expect(vi.getTimerCount()).toBe(0)
})
it('makes exactly two calls on persistent transient failure and releases the pool', async () => {
const failure = Object.assign(new Error('Unavailable'), { status: 503 })
const f = fixture(async function* (input) { yield { type: 'status', requestId: input.requestId, message: 'starting' }; throw failure })
const result = expect(f.start()).rejects.toBe(failure)
await vi.runAllTimersAsync()
await result
expect(f.runtime.run).toHaveBeenCalledTimes(2)
expect(f.release).toHaveBeenCalledTimes(2)
expect(f.dispose).toHaveBeenCalledOnce()
await expect(f.pool.run(async () => true)).resolves.toBe(true)
expect(vi.getTimerCount()).toBe(0)
})
it.each([
Object.assign(new Error('Unauthorized'), { status: 401 }),
Object.assign(new Error('Forbidden'), { status: 403 }),
Object.assign(new Error('Not implemented'), { status: 501 }),
Object.assign(new Error('Insufficient quota'), { status: 429 }),
Object.assign(new Error('Context length exceeded'), { status: 503 }),
Object.assign(new Error('Invalid API key'), { status: 429 }),
new Error('模型接口请求超时', { cause: new TypeError('terminated') }),
new DOMException('aborted', 'AbortError'),
new Error('模型接口返回了无效的流式 JSON'),
new Error('Provider unavailable'),
new Error('Configuration invalid'),
new TypeError('fetch failed', { cause: Object.assign(new Error('bad host'), { code: 'ENOTFOUND' }) }),
new TypeError('fetch failed', { cause: Object.assign(new Error('bad certificate'), { code: 'CERT_HAS_EXPIRED' }) }),
new Error('单次模型响应超过 100 KiB'),
new Error('Unrelated error mentioning 429')
])('does not retry nontransient errors, thrown or emitted: %s', async failure => {
for (const emitted of [false, true]) {
const f = fixture(async function* (input) {
if (emitted && !failure.cause) yield { type: 'error', status: 'failed', requestId: input.requestId, message: failure.message }
else throw failure
})
await expect(f.start()).rejects.toThrow(failure.message)
expect(f.runtime.run).toHaveBeenCalledOnce()
expect(f.release).toHaveBeenCalledOnce()
expect(f.dispose).toHaveBeenCalledOnce()
expect(vi.getTimerCount()).toBe(0)
}
})
it('cancels during the retry wait without dispatching another request', async () => {
const f = fixture(async function* (input) { yield { type: 'error', status: 'failed', requestId: input.requestId, message: 'terminated' } })
const result = expect(f.start()).rejects.toThrow()
await vi.advanceTimersByTimeAsync(100)
expect(f.release).toHaveBeenCalledOnce()
f.pool.cancelAll(new Error('User cancelled'))
await result
await vi.runAllTimersAsync()
expect(f.runtime.run).toHaveBeenCalledOnce()
expect(f.dispose).toHaveBeenCalledOnce()
await expect(f.pool.run(async () => true)).resolves.toBe(true)
expect(vi.getTimerCount()).toBe(0)
})
it('does not retry cancellation disguised as a transient stream error', async () => {
const f = fixture(async function* (input) {
yield { type: 'status', requestId: input.requestId, message: 'starting' }
f.pool.cancelAll(new Error('User cancelled'))
throw new TypeError('terminated')
})
await expect(f.start()).rejects.toThrow('User cancelled')
expect(f.runtime.run).toHaveBeenCalledOnce()
expect(f.statuses).toHaveBeenLastCalledWith(expect.any(String), 'cancelled', 'User cancelled')
})
it.each([false, true])('keeps a per-attempt timeout and never retries timeout (transient first: %s)', async transientFirst => {
let calls = 0
const f = fixture(async function* (input, signal) {
calls++
if (transientFirst && calls === 1) {
await new Promise(resolve => setTimeout(resolve, 900))
throw new TypeError('terminated')
}
await new Promise<void>((_, reject) => signal.addEventListener('abort', () => reject(new TypeError('terminated')), { once: true }))
yield { type: 'done', requestId: input.requestId }
}, 1)
const result = expect(f.start()).rejects.toThrow('超过 1 秒')
await vi.advanceTimersByTimeAsync(transientFirst ? 2300 : 900)
expect(calls).toBe(transientFirst ? 2 : 1)
expect(f.dispose).not.toHaveBeenCalled()
await vi.advanceTimersByTimeAsync(100)
await result
expect(f.release).toHaveBeenCalledTimes(transientFirst ? 2 : 1)
expect(f.dispose).toHaveBeenCalledOnce()
expect(vi.getTimerCount()).toBe(0)
})
it('does not retry a cancelled runtime error event with a transient-looking message', async () => {
const f = fixture(async function* (input) {
yield { type: 'error', status: 'cancelled', requestId: input.requestId, message: 'terminated' }
})
await expect(f.start()).rejects.toThrow('terminated')
expect(f.runtime.run).toHaveBeenCalledOnce()
expect(f.statuses).toHaveBeenLastCalledWith(expect.any(String), 'cancelled', 'terminated')
expect(f.release).toHaveBeenCalledOnce()
})
it('does not dispatch a retry if conversation cleanup fails', async () => {
const f = fixture(async function* (input) {
yield { type: 'error', status: 'failed', requestId: input.requestId, message: 'terminated' }
})
f.release.mockRejectedValueOnce(new Error('Cleanup failed'))
await expect(f.start()).rejects.toThrow('Cleanup failed')
expect(f.runtime.run).toHaveBeenCalledOnce()
expect(f.dispose).toHaveBeenCalledOnce()
await expect(f.pool.run(async () => true)).resolves.toBe(true)
expect(vi.getTimerCount()).toBe(0)
})
it.each([429, 500, 502, 503, 504])('retries actual direct-model HTTP %i with provider detail intact', async status => {
const detail = status === 504 ? 'Gateway Timeout' : 'Provider detail'
const fetcher = vi.fn<typeof fetch>()
.mockResolvedValueOnce(new Response(JSON.stringify({ error: { message: detail } }), { status }))
.mockResolvedValueOnce(new Response('data: {"choices":[{"delta":{"content":"success"}}]}\n\ndata: [DONE]\n\n'))
const runtime = new ModelAgentRuntime({ baseUrl: 'https://model.invalid', model: 'fake', authentication: 'none',
protocol: 'openai-chat-completions', toolProvider: noModelTools, fetcher })
const f = fixture(runtime.run.bind(runtime))
f.release.mockImplementation(runtime.releaseConversation.bind(runtime))
f.dispose.mockImplementation(runtime.dispose.bind(runtime))
const result = expect(f.start()).resolves.toBe('success')
await vi.runAllTimersAsync()
await result
expect(fetcher).toHaveBeenCalledTimes(2)
expect(f.statuses.mock.calls[0]?.[2]).toBe(detail)
expect(f.release).toHaveBeenCalledTimes(2)
expect(f.dispose).toHaveBeenCalledOnce()
})
it('retries an actual interrupted direct-model stream and retains its reported usage', async () => {
let reads = 0
const stream = new ReadableStream<Uint8Array>({ pull(controller) {
if (reads++ === 0) controller.enqueue(new TextEncoder().encode(
'data: {"choices":[{"delta":{"content":"discard me"}}],"usage":{"prompt_tokens":12,"completion_tokens":3}}\n\n'))
else controller.error(new TypeError('terminated'))
} })
const fetcher = vi.fn<typeof fetch>().mockResolvedValueOnce(new Response(stream))
.mockResolvedValueOnce(new Response('data: {"choices":[{"delta":{"content":"success"}}],"usage":{"prompt_tokens":14,"completion_tokens":4}}\n\ndata: [DONE]\n\n'))
const runtime = new ModelAgentRuntime({ baseUrl: 'https://model.invalid', model: 'fake', authentication: 'none',
protocol: 'openai-chat-completions', toolProvider: noModelTools, fetcher })
const f = fixture(runtime.run.bind(runtime))
f.release.mockImplementation(runtime.releaseConversation.bind(runtime))
f.dispose.mockImplementation(runtime.dispose.bind(runtime))
const result = expect(f.start()).resolves.toBe('success')
await vi.runAllTimersAsync()
await result
expect(fetcher).toHaveBeenCalledTimes(2)
expect(stream.locked).toBe(false)
expect(f.db.getTokenUsageSummary().systemTotals).toMatchObject({ callCount: 2, input: 26, output: 7 })
expect(f.release).toHaveBeenCalledTimes(2)
expect(f.dispose).toHaveBeenCalledOnce()
})
@@ -181,6 +181,7 @@ it('stops oversized runtime responses with a readable error while preserving res
const f = fixture() const f = fixture()
f.respond.mockReturnValue({ ...empty, summary: 'x'.repeat(110_000) }) f.respond.mockReturnValue({ ...empty, summary: 'x'.repeat(110_000) })
await expect(f.service(100).run(f.request)).rejects.toThrow('单次模型响应超过 100 KiB') await expect(f.service(100).run(f.request)).rejects.toThrow('单次模型响应超过 100 KiB')
expect(f.respond).toHaveBeenCalledOnce()
expect(f.db.listSupervisionResults()).toHaveLength(0) expect(f.db.listSupervisionResults()).toHaveLength(0)
expect(f.db.listSupervisionActivity()[0]!.reviewProgress).toMatchObject({ batches: 0, remainingSources: 1 }) expect(f.db.listSupervisionActivity()[0]!.reviewProgress).toMatchObject({ batches: 0, remainingSources: 1 })
const runId = f.db.listSupervisionActivity()[0]!.id const runId = f.db.listSupervisionActivity()[0]!.id
@@ -263,7 +264,7 @@ it('production factory normalizes two absent identities, then reuses exact candi
expect(f.respond).toHaveBeenCalledTimes(2) expect(f.respond).toHaveBeenCalledTimes(2)
}) })
it('retains candidate mappings across leaf batches, failed merge and resume without replaying leaves', async () => { it.each([false, true])('retains candidate mappings across leaf batches, failed merge and resume without replaying leaves (transient: %s)', async transient => {
const f = fixture('Atlas and Beacon. '.repeat(100)) const f = fixture('Atlas and Beacon. '.repeat(100))
f.db.saveSupervisionResult({ request: f.request, evidence: [], output: { ...empty, f.db.saveSupervisionResult({ request: f.request, evidence: [], output: { ...empty,
entities: ['Atlas', 'Beacon'].map(label => ({ id: label, label, description: label, sourceReferenceIds: [] })) } }) entities: ['Atlas', 'Beacon'].map(label => ({ id: label, label, description: label, sourceReferenceIds: [] })) } })
@@ -275,13 +276,15 @@ it('retains candidate mappings across leaf batches, failed merge and resume with
if (prompt.includes('These inputs are navigation summaries')) { if (prompt.includes('These inputs are navigation summaries')) {
expect(prompt).toContain('KNOWN ENTITIES:\n\n[]') expect(prompt).toContain('KNOWN ENTITIES:\n\n[]')
expect(prompt).toContain('"entities":[]') expect(prompt).toContain('"entities":[]')
if (failMerge) { failMerge = false; throw new Error('Merge interrupted') } if (failMerge) throw transient ? Object.assign(new Error('Merge interrupted'), { status: 503 }) : new Error('Merge interrupted')
return empty return empty
} }
const output = original(prompt) as { entities: Array<{ label: string }> } const output = original(prompt) as { entities: Array<{ label: string }> }
return { ...output, entities: output.entities.map(entity => ({ ...entity, persistedId: candidates.find(candidate => candidate.label === entity.label)!.id })) } return { ...output, entities: output.entities.map(entity => ({ ...entity, persistedId: candidates.find(candidate => candidate.label === entity.label)!.id })) }
}) })
await expect(f.service().run(f.request)).rejects.toThrow('Merge interrupted') await expect(f.service().run(f.request)).rejects.toThrow('Merge interrupted')
expect(f.respond.mock.calls.filter(([prompt]) => prompt.includes('These inputs are navigation summaries'))).toHaveLength(transient ? 2 : 1)
failMerge = false
const runId = f.db.listSupervisionActivity().find(item => item.status === 'failed')!.id const runId = f.db.listSupervisionActivity().find(item => item.status === 'failed')!.id
const saved = f.db.supervisionReviewStore().batches(runId) const saved = f.db.supervisionReviewStore().batches(runId)
expect(saved).toHaveLength(2) expect(saved).toHaveLength(2)
+83 -42
View File
@@ -20,7 +20,33 @@ type SupervisionModelDependencies = {
persistUsage: (event: RuntimeModelUsageEvent) => void persistUsage: (event: RuntimeModelUsageEvent) => void
} }
/** One bounded, tool-free text call that shares the supervisor concurrency pool. */ function isTransientSupervisionError(error: unknown): boolean {
let transient = false
const seen = new Set<unknown>()
for (let current = error; current && typeof current === 'object' && !seen.has(current);) {
seen.add(current)
const record = current as Record<string, unknown>
const message = typeof record.message === 'string' ? record.message : ''
const code = typeof record.code === 'string' ? record.code : ''
// Direct-model errors retain status; runtime events carry only the public message.
const status = record.status ?? record.statusCode ?? /\bHTTP\s+(\d{3})\b/iu.exec(message)?.[1]
const failureText = Number(status) === 504 ? message.replace(/\bgateway (?:timeout|timed out)\b/giu, '') : message
if (record.name === 'AbortError' || record.name === 'TimeoutError' ||
/^(?:ENOTFOUND|ERR_INVALID_URL|CERT_.*|.*CERTIFICATE.*|DEPTH_ZERO_SELF_SIGNED_CERT)$/u.test(code) ||
/abort|cancel|timed?\s*out|timeout|超时|取消|容量|context[_ ](?:length|window)|max[_ ]tokens|quota|billing|insufficient|capacity|api[_ -]?key|unauthori[sz]ed|forbidden|authentication|configuration/iu.test(`${code} ${failureText}`)) return false
if (status !== undefined) {
if (![429, 500, 502, 503, 504].includes(Number(status))) return false
transient = true
}
if (/^(?:ECONNRESET|ECONNREFUSED|EPIPE|EAI_AGAIN|ENETUNREACH|EHOSTUNREACH|UND_ERR_SOCKET)$/u.test(code) ||
/^(?:TypeError:\s*)?(?:terminated|fetch failed|network error|socket hang up)$/iu.test(message) ||
message === '模型接口流式响应意外中断' || /^rate limit (?:exceeded|reached)\b/iu.test(message)) transient = true
current = record.cause
}
return transient
}
/** One bounded, tool-free text call, with one transient retry in the supervisor pool. */
async function runSupervisionModel(dependencies: SupervisionModelDependencies, request: { async function runSupervisionModel(dependencies: SupervisionModelDependencies, request: {
prompt: string prompt: string
title: string title: string
@@ -36,49 +62,64 @@ async function runSupervisionModel(dependencies: SupervisionModelDependencies, r
return pool.run(async signal => { return pool.run(async signal => {
const timeoutSeconds = request.timeoutSeconds ?? settings?.supervisorOrganizeTimeoutSeconds ?? defaultSupervisionTimeoutSeconds const timeoutSeconds = request.timeoutSeconds ?? settings?.supervisorOrganizeTimeoutSeconds ?? defaultSupervisionTimeoutSeconds
const runtime = await resolveRuntime() const runtime = await resolveRuntime()
const controller = new AbortController() for (let attempt = 0; ; attempt++) {
const modelSignal = AbortSignal.any([signal, controller.signal, ...(request.signal ? [request.signal] : [])]) signal.throwIfAborted()
modelSignal.throwIfAborted() const controller = new AbortController()
let timeoutError: Error | undefined const modelSignal = AbortSignal.any([signal, controller.signal])
const timeout = setTimeout(() => { let timeoutError: Error | undefined
timeoutError = new Error(request.timeoutMessage(timeoutSeconds)) const timeout = setTimeout(() => {
controller.abort(timeoutError) timeoutError = new Error(request.timeoutMessage(timeoutSeconds))
}, timeoutSeconds * 1000) controller.abort(timeoutError)
const responseKiB = settings?.supervisionReview?.responseKiB ?? 1024 }, timeoutSeconds * 1000)
let output = '', completed = false const responseKiB = settings?.supervisionReview?.responseKiB ?? 1024
const requestId = randomUUID() let output = '', completed = false
const conversationId = `supervision:${requestId}` const requestId = randomUUID()
// Hidden task row so model usage can be attributed to supervision. const conversationId = `supervision:${requestId}`
database.createTask({ id: requestId, conversationId, title: request.title, instructions: request.instructions,
origin: 'assistant', visible: false })
try {
for await (const event of runtime.run({ requestId, conversationId, prompt: request.prompt },
modelSignal, async approval => { await request.authorizeTool(approval.toolName ?? approval.scopeKey); return 'deny' })) {
if (event.type === 'text') {
output += event.delta
if (Buffer.byteLength(output) > responseKiB * 1024) {
controller.abort(new Error(`单次模型响应超过 ${responseKiB} KiB。请在监督者设置中调高响应容量后继续;已保存批次会保留。`))
modelSignal.throwIfAborted()
}
}
if (event.type === 'model-usage') persistUsage(event)
if (event.type === 'tool') throw new Error('监督者只允许只读模型摘要,不允许工具调用')
if (event.type === 'error') throw new Error(event.message)
if (event.type === 'done') completed = true
}
modelSignal.throwIfAborted()
if (!completed) throw new Error('监督者模型未报告完成')
database.updateTaskStatus(requestId, 'completed')
return output
} catch (error) {
const failure = timeoutError ?? error
try { try {
database.updateTaskStatus(requestId, modelSignal.aborted ? 'cancelled' : 'failed', // Each attempt has its own hidden task so reported usage survives retries.
failure instanceof Error ? failure.message.slice(0, 2_000) : '监督者模型调用失败') database.createTask({ id: requestId, conversationId, title: request.title, instructions: request.instructions,
} catch { /* status bookkeeping must not mask the model failure */ } origin: 'assistant', visible: false })
throw failure for await (const event of runtime.run({ requestId, conversationId, prompt: request.prompt },
modelSignal, async approval => { await request.authorizeTool(approval.toolName ?? approval.scopeKey); return 'deny' })) {
if (event.type === 'text') {
output += event.delta
if (Buffer.byteLength(output) > responseKiB * 1024) {
controller.abort(new Error(`单次模型响应超过 ${responseKiB} KiB。请在监督者设置中调高响应容量后继续;已保存批次会保留。`))
modelSignal.throwIfAborted()
}
}
if (event.type === 'model-usage') persistUsage(event)
if (event.type === 'tool') throw new Error('监督者只允许只读模型摘要,不允许工具调用')
if (event.type === 'error') {
const error = new Error(event.message)
if (event.status === 'cancelled') controller.abort(error)
throw error
}
if (event.type === 'done') completed = true
}
modelSignal.throwIfAborted()
if (!completed) throw new Error('监督者模型未报告完成')
database.updateTaskStatus(requestId, 'completed')
return output
} catch (error) {
const failure = timeoutError ?? (modelSignal.aborted ? modelSignal.reason : error)
try {
database.updateTaskStatus(requestId, modelSignal.aborted ? 'cancelled' : 'failed',
failure instanceof Error ? failure.message.slice(0, 2_000) : '监督者模型调用失败')
} catch { /* status bookkeeping must not mask the model failure */ }
if (attempt > 0 || modelSignal.aborted || !isTransientSupervisionError(failure)) throw failure
} finally {
clearTimeout(timeout)
controller.abort()
await runtime.releaseConversation?.(conversationId)
}
signal.throwIfAborted()
await new Promise<void>((resolve, reject) => {
const abort = () => { clearTimeout(wait); reject(signal.reason) }
const wait = setTimeout(() => { signal.removeEventListener('abort', abort); resolve() }, 500)
signal.addEventListener('abort', abort, { once: true })
})
} }
finally { clearTimeout(timeout); await runtime.releaseConversation?.(conversationId) }
}, request.signal) }, request.signal)
} }