fix: retry transient supervisor model failures once
This commit is contained in:
@@ -10,6 +10,8 @@
|
|||||||
|
|
||||||
回顾获得执行位置后,读取一次连接配置;同次执行中的批次、输出重试、汇总、故事与经验阶段保持该模型。修改监督者选择、应用默认模型或连接配置,仅影响后续执行。继续暂停或失败的回顾、单独重试故事、生成或重试建议时重新读取当前选择;跨重启不保存模型或凭据快照。显式选择的连接被删除、改为图片协议或缺少所需 API Key 时,执行报错并提示修改设置,不自动换成默认模型。
|
回顾获得执行位置后,读取一次连接配置;同次执行中的批次、输出重试、汇总、故事与经验阶段保持该模型。修改监督者选择、应用默认模型或连接配置,仅影响后续执行。继续暂停或失败的回顾、单独重试故事、生成或重试建议时重新读取当前选择;跨重启不保存模型或凭据快照。显式选择的连接被删除、改为图片协议或缺少所需 API Key 时,执行报错并提示修改设置,不自动换成默认模型。
|
||||||
|
|
||||||
|
对应 FR-S6。监督模型遇到可识别的临时失败时,统一等待 500ms 后重试一次:HTTP 429、500/502/503/504,以及短暂网络故障或响应流中断。取消、模型超时、认证、配置、额度或容量错误直接停止;无法确认可恢复的错误也不重试。失败尝试的部分正文丢弃,已报告用量和已保存批次保留。叶子输出校验后的缩块重试、导航输出重试保持原规则;请求生命周期见[模型阶段超时与调度](./technical-design.md#模型阶段超时与调度)。
|
||||||
|
|
||||||
## 整体逻辑架构
|
## 整体逻辑架构
|
||||||
|
|
||||||
报告、监督叶子与导航整理共用监督专用 FIFO 并发池,普通聊天不入池;监督服务外层 run 仍串行。报告在排队前冻结超时,获槽后领取,实际模型请求前校验并刷新租约;报告提交或无变化后先释放槽位,再执行监督。监督超时在 run 创建时保存,节点获槽并解析 Runtime 后计时。阶段超时、租约、传输时限、后台检查与活动轮询各自独立,取值及生效规则见[当前设置合同](./review-scheduling-design.md#当前设置合同)。
|
报告、监督叶子与导航整理共用监督专用 FIFO 并发池,普通聊天不入池;监督服务外层 run 仍串行。报告在排队前冻结超时,获槽后领取,实际模型请求前校验并刷新租约;报告提交或无变化后先释放槽位,再执行监督。监督超时在 run 创建时保存,节点获槽并解析 Runtime 后计时。阶段超时、租约、传输时限、后台检查与活动轮询各自独立,取值及生效规则见[当前设置合同](./review-scheduling-design.md#当前设置合同)。
|
||||||
|
|||||||
@@ -4,6 +4,17 @@
|
|||||||
|
|
||||||
当前记录以已验证生产行为为准。监督者尚未覆盖全部 user stories。
|
当前记录以已验证生产行为为准。监督者尚未覆盖全部 user stories。
|
||||||
|
|
||||||
|
## 2026-10-05 监督模型临时失败重试
|
||||||
|
|
||||||
|
对应 FR-S6。`runSupervisionModel` 已接入一次可取消的临时失败重试,覆盖监督者各模型阶段;错误分类、清理及每次尝试的超时规则见[技术设计](./technical-design.md#模型阶段超时与调度)。直连文本请求保留 HTTP status,并交出失败流中已报告的用量。OpenCode/Continue、直连传输重试和叶子/导航输出重试未改。监督者仍在桌面 Main 直连模型,不经过远程 Agent 或模型桥。
|
||||||
|
|
||||||
|
验证结果:
|
||||||
|
|
||||||
|
- `npx vitest run src/main/assistant/supervision-model-retry.test.ts src/main/assistant/supervision-production.test.ts src/main/assistant/supervision-review.test.ts src/main/assistant/supervision-model-pool.test.ts src/main/agent/model-runtime.test.ts`:5 个文件,177 项通过,1 项原有跳过。覆盖 429、可恢复 5xx、部分正文后 terminated、连续失败最多两次包装层调用、认证/配置/容量/超时不重试、等待中取消、流和会话清理、用量保留,以及已保存叶子在失败和继续后不重算。
|
||||||
|
- `npx tsc --noEmit -p tsconfig.node.json`、`npx tsc --noEmit -p tsconfig.agent.json`、4 个改动 TypeScript 文件的定向 ESLint 和 `git diff --check` 通过。
|
||||||
|
|
||||||
|
请求验证使用生产 `ModelAgentRuntime` 和注入的假 fetch/响应流,数据保存使用内存 SQLite;外部模型调用为 0。未运行全量测试、真实模型或远程 Host 验证,未提交或推送。
|
||||||
|
|
||||||
## 2026-10-05 后端阶段审查 A03/A04/A05
|
## 2026-10-05 后端阶段审查 A03/A04/A05
|
||||||
|
|
||||||
对应 SL-5、SL-8、FR-S6、FR-S10。生产候选读取现在将批次项目和跨项目设置传到只读 Worker;最终候选读取返回后检查暂停/取消;故事归属和拆分在原写事务内拒绝已移除或合并的目标。存储、停止与重试规则见[技术设计](./technical-design.md#单次执行与取消)。没有 schema 变更,原始消息、历史结果及人工确认字段保留。
|
对应 SL-5、SL-8、FR-S6、FR-S10。生产候选读取现在将批次项目和跨项目设置传到只读 Worker;最终候选读取返回后检查暂停/取消;故事归属和拆分在原写事务内拒绝已移除或合并的目标。存储、停止与重试规则见[技术设计](./technical-design.md#单次执行与取消)。没有 schema 变更,原始消息、历史结果及人工确认字段保留。
|
||||||
@@ -156,7 +167,7 @@ FR-S13 已接通监督者“设置 → 模型”、应用设置持久化和 Main
|
|||||||
|
|
||||||
- 叶子批次输出不可用(无效 JSON、结构不符、引用不存在的来源或实体)时,在同一段待处理正文上缩小到 1/2、再到 1/4(不少于 500 字符)重试,最多 2 次;缩小只是先处理前一部分,其余正文留在待处理里,不跳过、不截断。
|
- 叶子批次输出不可用(无效 JSON、结构不符、引用不存在的来源或实体)时,在同一段待处理正文上缩小到 1/2、再到 1/4(不少于 500 字符)重试,最多 2 次;缩小只是先处理前一部分,其余正文留在待处理里,不跳过、不截断。
|
||||||
- 导航合并输出不可用时,用相同输入再问 1 次。
|
- 导航合并输出不可用时,用相同输入再问 1 次。
|
||||||
- 提供方错误、超时、取消和响应超出容量不重试,仍按原来的方式失败并保留已保存批次。
|
- 当时提供方错误、超时、取消和响应超出容量不重试,失败保留已保存批次。2026-10-05 起临时提供方错误采用上文的一次重试;超时、取消和响应超出容量仍不重试。
|
||||||
|
|
||||||
改后结果(DeepSeek `deepseek-flash`):
|
改后结果(DeepSeek `deepseek-flash`):
|
||||||
|
|
||||||
|
|||||||
@@ -12,6 +12,8 @@
|
|||||||
|
|
||||||
Main 的手动 IPC 和心跳完成回调均使用 `createProductionSupervisorService`,该工厂连接 `SupervisorService`、`SupervisionReviewStore`、共享模型池及无工具 Runtime。生产路径不再调用旧的 20 会话 × 20 消息 collector,也不使用旧服务分支的 100 条 / 48,000 字符裁剪。旧的有界服务注入方式仍供原有测试和实验脚本使用;心跳报告本身继续使用原报告预算。
|
Main 的手动 IPC 和心跳完成回调均使用 `createProductionSupervisorService`,该工厂连接 `SupervisorService`、`SupervisionReviewStore`、共享模型池及无工具 Runtime。生产路径不再调用旧的 20 会话 × 20 消息 collector,也不使用旧服务分支的 100 条 / 48,000 字符裁剪。旧的有界服务注入方式仍供原有测试和实验脚本使用;心跳报告本身继续使用原报告预算。
|
||||||
|
|
||||||
|
2026-10-05 起,监督模型包装层对可识别的临时失败统一重试一次,包含 429、可恢复 5xx 和网络/响应流中断;取消、模型超时、配置及容量错误不重试。每次尝试独立计时,等待可取消,成功批次保留。具体错误与生命周期合同见[技术设计](./technical-design.md#模型阶段超时与调度);下文旧草案中“不增加节点网络重试”的描述已由此规则替代。
|
||||||
|
|
||||||
### 已接入的调度与存储
|
### 已接入的调度与存储
|
||||||
|
|
||||||
schema 47 增加运行配置、来源清单、成功批次、导航节点四张表和一个实时来源视图。迁移只创建新结构,保留此前迁移、结果及用户数据。清单只保存来源身份、版本、长度及增量起点;正文随后按主键和有界 `substr` 读取,派发顺序仍由 `(project_id, conversation_id, sequence, source)` 决定。
|
schema 47 增加运行配置、来源清单、成功批次、导航节点四张表和一个实时来源视图。迁移只创建新结构,保留此前迁移、结果及用户数据。清单只保存来源身份、版本、长度及增量起点;正文随后按主键和有界 `substr` 读取,派发顺序仍由 `(project_id, conversation_id, sequence, source)` 决定。
|
||||||
@@ -50,7 +52,7 @@ schema 47 增加运行配置、来源清单、成功批次、导航节点四张
|
|||||||
| `supervisionReview.executionSeconds` | 300 | 30..3600 | 旧设置兼容字段,执行时忽略,设置页不再展示;恢复旧 run 同样不按此值暂停 |
|
| `supervisionReview.executionSeconds` | 300 | 30..3600 | 旧设置兼容字段,执行时忽略,设置页不再展示;恢复旧 run 同样不按此值暂停 |
|
||||||
| `supervisionReview.crossProject` | 关闭 | 开关 | 创建回顾时冻结;关闭时每批只以本项目已有实体为候选,也不生成跨项目故事;开启后候选来自全部项目,故事归属可建立跨项目故事 |
|
| `supervisionReview.crossProject` | 关闭 | 开关 | 创建回顾时冻结;关闭时每批只以本项目已有实体为候选,也不生成跨项目故事;开启后候选来自全部项目,故事归属可建立跨项目故事 |
|
||||||
| `supervisionReview.storyThreadEvents` | 20 | 4..500 | 创建回顾时冻结;功能的事件数低于此值时,模型提出的子线索并入功能 |
|
| `supervisionReview.storyThreadEvents` | 20 | 4..500 | 创建回顾时冻结;功能的事件数低于此值时,模型提出的子线索并入功能 |
|
||||||
| `supervisorOrganizeTimeoutSeconds` | 240 | 30..600 | 创建回顾时冻结,节点获槽并解析 Runtime 后计时 |
|
| `supervisorOrganizeTimeoutSeconds` | 240 | 30..600 | 创建回顾时冻结,节点获槽并解析 Runtime 后,每次模型尝试独立计时 |
|
||||||
| `heartbeatReportTimeoutSeconds` | 240 | 30..600 | 维持报告发起时冻结及既有租约规则 |
|
| `heartbeatReportTimeoutSeconds` | 240 | 30..600 | 维持报告发起时冻结及既有租约规则 |
|
||||||
| `supervisorModelConcurrency` | 1 | 1..4 | 回顾保存接纳并发;共享池上限仍随应用设置实时变化,实际执行取较小值 |
|
| `supervisorModelConcurrency` | 1 | 1..4 | 回顾保存接纳并发;共享池上限仍随应用设置实时变化,实际执行取较小值 |
|
||||||
|
|
||||||
|
|||||||
@@ -55,9 +55,13 @@ schema 43 在结果上增加 `graph_snapshot_json`,保存当次实体名称、
|
|||||||
|
|
||||||
监督整理超时由 run 创建时冻结;共享模型池仍采用实时设置上限。`supervision:pause` 取消该 run 的排队和在途工作,`supervision:resume` 继续已保存批次;监督整理输出在流中按 `supervisionReview.responseKiB` 检查响应容量,默认 1024 KiB,可在设置页调整为 100 至 16384 KiB。该容量每次请求读取当前设置,继续旧运行也采用新值;超限保留成功批次并提示调高后继续。生成摘要及事实内容不设独立短字符限制,监督旧摘要读取保留全文,具体边界见[生产接线](./review-scheduling-design.md#0-生产接线与剩余边界)。以下报告领取和租约规则继续适用。
|
监督整理超时由 run 创建时冻结;共享模型池仍采用实时设置上限。`supervision:pause` 取消该 run 的排队和在途工作,`supervision:resume` 继续已保存批次;监督整理输出在流中按 `supervisionReview.responseKiB` 检查响应容量,默认 1024 KiB,可在设置页调整为 100 至 16384 KiB。该容量每次请求读取当前设置,继续旧运行也采用新值;超限保留成功批次并提示调高后继续。生成摘要及事实内容不设独立短字符限制,监督旧摘要读取保留全文,具体边界见[生产接线](./review-scheduling-design.md#0-生产接线与剩余边界)。以下报告领取和租约规则继续适用。
|
||||||
|
|
||||||
手动和自动监督均在桌面 Main 调用生产工厂,Runtime 解析请求不携带项目、Runtime 选择或执行空间,也不再注入 `workMode`。`resolveRequestRuntime` 返回桌面默认 Runtime,不进入 SSH 项目或 `selectedRuntimes.getRuntime` 分支;内部分析使用无工具构造,回顾 scope 只筛选证据。摘要校验、合并和持久化在 Main 完成,single-flight、取消及 UI 控制不改变 `gbagent`、远端 Runtime 启动或桌面到 Agent 协议。
|
手动和自动监督均在桌面 Main 调用生产工厂,由 `resolveSupervisorRuntime` 解析上文的直连模型,不进入 SSH 项目或 `selectedRuntimes.getRuntime` 分支;内部分析使用无工具构造,回顾 scope 只筛选证据。摘要校验、合并和持久化在 Main 完成,single-flight、取消及 UI 控制不改变 `gbagent`、远端 Runtime 启动或桌面到 Agent 协议。
|
||||||
|
|
||||||
`ApplicationSettingsStore` 持久化报告和整理模型超时,均为 30..600 整数秒,默认 240。报告排队前冻结,监督创建 run 时冻结;计时从获槽及 Runtime 解析后开始,覆盖模型执行及其内部重试。流结束校验 signal 和完成事件,finally 释放临时会话,心跳报告保持原有 100KB 上限,监督整理使用上文的可调响应容量。采集、排队、保存和清理不计入模型超时。
|
`ApplicationSettingsStore` 持久化报告和整理模型超时,均为 30..600 整数秒,默认 240。报告排队前冻结,监督创建 run 时冻结;计时从获槽及 Runtime 解析后开始,覆盖单次 Runtime 执行及其内部重试。监督包装层的第二次尝试重新按同一秒数计时,500ms 等待和临时会话清理不计入该时限。流结束校验 signal 和完成事件,finally 释放临时会话,心跳报告保持原有 100KB 上限,监督整理使用上文的可调响应容量。采集、排队、保存和清理不计入模型超时。
|
||||||
|
|
||||||
|
`runSupervisionModel` 为提取、导航、故事、经验和建议共用一次临时错误重试,错误范围见[逻辑规则](./logic-design.md#监督者模型选择)。每次尝试使用新的 request/conversation ID 和隐藏任务,正文缓冲单独创建;用量按各次任务保存。失败后先清理计时器并释放临时会话,再等待;等待继续持有监督池槽位,受父 signal 和池取消控制,结束或失败后释放槽位。`error` 事件的 `status: cancelled` 优先于错误文字;抛出的结构化错误检查 status、statusCode 和 cause,字符串只识别明确的 HTTP 状态或已知网络/流中断格式。
|
||||||
|
|
||||||
|
直连文本请求保留 HTTP 错误的 status,并在响应流失败前交出已收到的用量;没有提供方用量时不估算补记。重试决策只在监督包装层,直连传输原有重试、OpenCode 和 Continue 策略不变。429 使用同一 500ms 等待,不新增 Retry-After、UI 或配置项。
|
||||||
|
|
||||||
`supervisorModelConcurrency` 为 1 至 4 的整数,默认 1。报告和监督整理共用 Main 内 `SupervisionModelPool` FIFO 池,普通聊天不入池;降低上限不终止在途请求。报告槽位从领取前持有至报告完成或无变化提交后,在下游监督开始前释放,避免并发 1 时相互等待。监督服务外层运行仍串行;该池不是完整分块调度器,也不提供跨聊天与监督的提供商全局并发、RPM、TPM 或 Retry-After 控制。
|
`supervisorModelConcurrency` 为 1 至 4 的整数,默认 1。报告和监督整理共用 Main 内 `SupervisionModelPool` FIFO 池,普通聊天不入池;降低上限不终止在途请求。报告槽位从领取前持有至报告完成或无变化提交后,在下游监督开始前释放,避免并发 1 时相互等待。监督服务外层运行仍串行;该池不是完整分块调度器,也不提供跨聊天与监督的提供商全局并发、RPM、TPM 或 Retry-After 控制。
|
||||||
|
|
||||||
|
|||||||
@@ -3168,9 +3168,9 @@ export class ModelAgentRuntime implements AgentRuntime {
|
|||||||
} catch {
|
} catch {
|
||||||
detail = undefined
|
detail = undefined
|
||||||
}
|
}
|
||||||
throw new Error(
|
throw Object.assign(new Error(
|
||||||
detail ?? `模型接口请求失败(HTTP ${response.status})`
|
detail ?? `模型接口请求失败(HTTP ${response.status})`
|
||||||
)
|
), { status: response.status })
|
||||||
}
|
}
|
||||||
|
|
||||||
let receivedStop = false
|
let receivedStop = false
|
||||||
@@ -3208,6 +3208,8 @@ export class ModelAgentRuntime implements AgentRuntime {
|
|||||||
}
|
}
|
||||||
return { answer, usage }
|
return { answer, usage }
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
const usageEvent = createUsageEvent(request.requestId, anthropic ? 'anthropic' : 'openai', this.options.model, usage)
|
||||||
|
if (usageEvent) yield usageEvent
|
||||||
return normalizeRequestError(error, modelRequest.timedOut())
|
return normalizeRequestError(error, modelRequest.timedOut())
|
||||||
} finally {
|
} finally {
|
||||||
modelRequest.clear()
|
modelRequest.clear()
|
||||||
|
|||||||
@@ -0,0 +1,231 @@
|
|||||||
|
import { afterEach, expect, it, vi } from 'vitest'
|
||||||
|
import type { AgentRuntime } from '../agent/runtime'
|
||||||
|
import { ModelAgentRuntime, noModelTools } from '../agent/model-runtime'
|
||||||
|
import { AssistantDatabase } from './assistant-database'
|
||||||
|
import { SupervisionModelPool } from './supervision-model-pool'
|
||||||
|
import { createProductionSuggestionPhraser } from './supervision-production'
|
||||||
|
|
||||||
|
const cleanups: Array<() => void> = []
|
||||||
|
afterEach(() => {
|
||||||
|
for (const cleanup of cleanups.splice(0)) cleanup()
|
||||||
|
vi.useRealTimers()
|
||||||
|
})
|
||||||
|
|
||||||
|
function fixture(run: AgentRuntime['run'], timeoutSeconds = 30) {
|
||||||
|
vi.useFakeTimers()
|
||||||
|
const db = new AssistantDatabase(':memory:')
|
||||||
|
db.initialize(process.cwd())
|
||||||
|
const pool = new SupervisionModelPool()
|
||||||
|
const release = vi.fn<NonNullable<AgentRuntime['releaseConversation']>>(async () => {})
|
||||||
|
const dispose = vi.fn(async () => {})
|
||||||
|
const runtime = { run: vi.fn(run), releaseConversation: release, dispose } as unknown as AgentRuntime
|
||||||
|
const statuses = vi.spyOn(db, 'updateTaskStatus')
|
||||||
|
const phrase = createProductionSuggestionPhraser(db, async () => ({ supervisorModelConcurrency: 1,
|
||||||
|
supervisorOrganizeTimeoutSeconds: timeoutSeconds }), async () => runtime, pool)
|
||||||
|
cleanups.push(() => { pool.dispose(); db.close() })
|
||||||
|
return { db, pool, runtime, release, dispose, statuses,
|
||||||
|
start: () => phrase({ systemInstruction: 'Suggest', outputContract: '{}', candidates: [] }) }
|
||||||
|
}
|
||||||
|
|
||||||
|
it.each([
|
||||||
|
Object.assign(new Error('Slow down'), { status: 429 }),
|
||||||
|
Object.assign(new Error('Gateway unavailable'), { statusCode: 502 }),
|
||||||
|
new TypeError('fetch failed', { cause: Object.assign(new Error('socket closed'), { code: 'ECONNRESET' }) }),
|
||||||
|
new TypeError('terminated'),
|
||||||
|
'模型接口请求失败(HTTP 503)',
|
||||||
|
'Gateway Timeout (HTTP 504)',
|
||||||
|
'TypeError: terminated',
|
||||||
|
'模型接口流式响应意外中断',
|
||||||
|
'Rate limit exceeded'
|
||||||
|
])('retries a transient failure once with fresh text, usage identities and cleanup: %s', async failure => {
|
||||||
|
let calls = 0
|
||||||
|
const finalized = vi.fn()
|
||||||
|
const f = fixture(async function* (input, signal) {
|
||||||
|
calls++
|
||||||
|
try {
|
||||||
|
yield { type: 'model-usage', requestId: input.requestId, callId: 'provider-call', runtime: 'model', provider: 'openai',
|
||||||
|
model: 'fake', inputTokens: 10, outputTokens: 2, cacheReadTokens: 0, cacheWriteTokens: 0 }
|
||||||
|
if (calls === 1) {
|
||||||
|
yield { type: 'text', requestId: input.requestId, delta: 'discard partial {' }
|
||||||
|
if (typeof failure === 'string') yield { type: 'error', status: 'failed', requestId: input.requestId, message: failure }
|
||||||
|
else throw failure
|
||||||
|
} else {
|
||||||
|
expect(f.release).toHaveBeenCalledTimes(1)
|
||||||
|
expect(signal.aborted).toBe(false)
|
||||||
|
yield { type: 'text', requestId: input.requestId, delta: 'clean result' }
|
||||||
|
yield { type: 'done', requestId: input.requestId }
|
||||||
|
}
|
||||||
|
} finally { finalized() }
|
||||||
|
})
|
||||||
|
const result = f.start()
|
||||||
|
await vi.advanceTimersByTimeAsync(499)
|
||||||
|
expect(calls).toBe(1)
|
||||||
|
expect(f.release).toHaveBeenCalledTimes(1)
|
||||||
|
await vi.advanceTimersByTimeAsync(1)
|
||||||
|
await expect(result).resolves.toBe('clean result')
|
||||||
|
expect(calls).toBe(2)
|
||||||
|
expect(finalized).toHaveBeenCalledTimes(2)
|
||||||
|
expect(f.release).toHaveBeenCalledTimes(2)
|
||||||
|
expect(new Set(f.release.mock.calls.map(args => args[0])).size).toBe(2)
|
||||||
|
expect(f.dispose).toHaveBeenCalledOnce()
|
||||||
|
expect(f.statuses.mock.calls.map(args => args[1])).toEqual(['failed', 'completed'])
|
||||||
|
expect(f.db.getTokenUsageSummary().systemTotals).toMatchObject({ callCount: 2, input: 20, output: 4 })
|
||||||
|
expect(f.db.listTasks()).toEqual([])
|
||||||
|
await expect(f.pool.run(async () => 'released')).resolves.toBe('released')
|
||||||
|
expect(vi.getTimerCount()).toBe(0)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('makes exactly two calls on persistent transient failure and releases the pool', async () => {
|
||||||
|
const failure = Object.assign(new Error('Unavailable'), { status: 503 })
|
||||||
|
const f = fixture(async function* (input) { yield { type: 'status', requestId: input.requestId, message: 'starting' }; throw failure })
|
||||||
|
const result = expect(f.start()).rejects.toBe(failure)
|
||||||
|
await vi.runAllTimersAsync()
|
||||||
|
await result
|
||||||
|
expect(f.runtime.run).toHaveBeenCalledTimes(2)
|
||||||
|
expect(f.release).toHaveBeenCalledTimes(2)
|
||||||
|
expect(f.dispose).toHaveBeenCalledOnce()
|
||||||
|
await expect(f.pool.run(async () => true)).resolves.toBe(true)
|
||||||
|
expect(vi.getTimerCount()).toBe(0)
|
||||||
|
})
|
||||||
|
|
||||||
|
it.each([
|
||||||
|
Object.assign(new Error('Unauthorized'), { status: 401 }),
|
||||||
|
Object.assign(new Error('Forbidden'), { status: 403 }),
|
||||||
|
Object.assign(new Error('Not implemented'), { status: 501 }),
|
||||||
|
Object.assign(new Error('Insufficient quota'), { status: 429 }),
|
||||||
|
Object.assign(new Error('Context length exceeded'), { status: 503 }),
|
||||||
|
Object.assign(new Error('Invalid API key'), { status: 429 }),
|
||||||
|
new Error('模型接口请求超时', { cause: new TypeError('terminated') }),
|
||||||
|
new DOMException('aborted', 'AbortError'),
|
||||||
|
new Error('模型接口返回了无效的流式 JSON'),
|
||||||
|
new Error('Provider unavailable'),
|
||||||
|
new Error('Configuration invalid'),
|
||||||
|
new TypeError('fetch failed', { cause: Object.assign(new Error('bad host'), { code: 'ENOTFOUND' }) }),
|
||||||
|
new TypeError('fetch failed', { cause: Object.assign(new Error('bad certificate'), { code: 'CERT_HAS_EXPIRED' }) }),
|
||||||
|
new Error('单次模型响应超过 100 KiB'),
|
||||||
|
new Error('Unrelated error mentioning 429')
|
||||||
|
])('does not retry nontransient errors, thrown or emitted: %s', async failure => {
|
||||||
|
for (const emitted of [false, true]) {
|
||||||
|
const f = fixture(async function* (input) {
|
||||||
|
if (emitted && !failure.cause) yield { type: 'error', status: 'failed', requestId: input.requestId, message: failure.message }
|
||||||
|
else throw failure
|
||||||
|
})
|
||||||
|
await expect(f.start()).rejects.toThrow(failure.message)
|
||||||
|
expect(f.runtime.run).toHaveBeenCalledOnce()
|
||||||
|
expect(f.release).toHaveBeenCalledOnce()
|
||||||
|
expect(f.dispose).toHaveBeenCalledOnce()
|
||||||
|
expect(vi.getTimerCount()).toBe(0)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('cancels during the retry wait without dispatching another request', async () => {
|
||||||
|
const f = fixture(async function* (input) { yield { type: 'error', status: 'failed', requestId: input.requestId, message: 'terminated' } })
|
||||||
|
const result = expect(f.start()).rejects.toThrow()
|
||||||
|
await vi.advanceTimersByTimeAsync(100)
|
||||||
|
expect(f.release).toHaveBeenCalledOnce()
|
||||||
|
f.pool.cancelAll(new Error('User cancelled'))
|
||||||
|
await result
|
||||||
|
await vi.runAllTimersAsync()
|
||||||
|
expect(f.runtime.run).toHaveBeenCalledOnce()
|
||||||
|
expect(f.dispose).toHaveBeenCalledOnce()
|
||||||
|
await expect(f.pool.run(async () => true)).resolves.toBe(true)
|
||||||
|
expect(vi.getTimerCount()).toBe(0)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('does not retry cancellation disguised as a transient stream error', async () => {
|
||||||
|
const f = fixture(async function* (input) {
|
||||||
|
yield { type: 'status', requestId: input.requestId, message: 'starting' }
|
||||||
|
f.pool.cancelAll(new Error('User cancelled'))
|
||||||
|
throw new TypeError('terminated')
|
||||||
|
})
|
||||||
|
await expect(f.start()).rejects.toThrow('User cancelled')
|
||||||
|
expect(f.runtime.run).toHaveBeenCalledOnce()
|
||||||
|
expect(f.statuses).toHaveBeenLastCalledWith(expect.any(String), 'cancelled', 'User cancelled')
|
||||||
|
})
|
||||||
|
|
||||||
|
it.each([false, true])('keeps a per-attempt timeout and never retries timeout (transient first: %s)', async transientFirst => {
|
||||||
|
let calls = 0
|
||||||
|
const f = fixture(async function* (input, signal) {
|
||||||
|
calls++
|
||||||
|
if (transientFirst && calls === 1) {
|
||||||
|
await new Promise(resolve => setTimeout(resolve, 900))
|
||||||
|
throw new TypeError('terminated')
|
||||||
|
}
|
||||||
|
await new Promise<void>((_, reject) => signal.addEventListener('abort', () => reject(new TypeError('terminated')), { once: true }))
|
||||||
|
yield { type: 'done', requestId: input.requestId }
|
||||||
|
}, 1)
|
||||||
|
const result = expect(f.start()).rejects.toThrow('超过 1 秒')
|
||||||
|
await vi.advanceTimersByTimeAsync(transientFirst ? 2300 : 900)
|
||||||
|
expect(calls).toBe(transientFirst ? 2 : 1)
|
||||||
|
expect(f.dispose).not.toHaveBeenCalled()
|
||||||
|
await vi.advanceTimersByTimeAsync(100)
|
||||||
|
await result
|
||||||
|
expect(f.release).toHaveBeenCalledTimes(transientFirst ? 2 : 1)
|
||||||
|
expect(f.dispose).toHaveBeenCalledOnce()
|
||||||
|
expect(vi.getTimerCount()).toBe(0)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('does not retry a cancelled runtime error event with a transient-looking message', async () => {
|
||||||
|
const f = fixture(async function* (input) {
|
||||||
|
yield { type: 'error', status: 'cancelled', requestId: input.requestId, message: 'terminated' }
|
||||||
|
})
|
||||||
|
await expect(f.start()).rejects.toThrow('terminated')
|
||||||
|
expect(f.runtime.run).toHaveBeenCalledOnce()
|
||||||
|
expect(f.statuses).toHaveBeenLastCalledWith(expect.any(String), 'cancelled', 'terminated')
|
||||||
|
expect(f.release).toHaveBeenCalledOnce()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('does not dispatch a retry if conversation cleanup fails', async () => {
|
||||||
|
const f = fixture(async function* (input) {
|
||||||
|
yield { type: 'error', status: 'failed', requestId: input.requestId, message: 'terminated' }
|
||||||
|
})
|
||||||
|
f.release.mockRejectedValueOnce(new Error('Cleanup failed'))
|
||||||
|
await expect(f.start()).rejects.toThrow('Cleanup failed')
|
||||||
|
expect(f.runtime.run).toHaveBeenCalledOnce()
|
||||||
|
expect(f.dispose).toHaveBeenCalledOnce()
|
||||||
|
await expect(f.pool.run(async () => true)).resolves.toBe(true)
|
||||||
|
expect(vi.getTimerCount()).toBe(0)
|
||||||
|
})
|
||||||
|
|
||||||
|
it.each([429, 500, 502, 503, 504])('retries actual direct-model HTTP %i with provider detail intact', async status => {
|
||||||
|
const detail = status === 504 ? 'Gateway Timeout' : 'Provider detail'
|
||||||
|
const fetcher = vi.fn<typeof fetch>()
|
||||||
|
.mockResolvedValueOnce(new Response(JSON.stringify({ error: { message: detail } }), { status }))
|
||||||
|
.mockResolvedValueOnce(new Response('data: {"choices":[{"delta":{"content":"success"}}]}\n\ndata: [DONE]\n\n'))
|
||||||
|
const runtime = new ModelAgentRuntime({ baseUrl: 'https://model.invalid', model: 'fake', authentication: 'none',
|
||||||
|
protocol: 'openai-chat-completions', toolProvider: noModelTools, fetcher })
|
||||||
|
const f = fixture(runtime.run.bind(runtime))
|
||||||
|
f.release.mockImplementation(runtime.releaseConversation.bind(runtime))
|
||||||
|
f.dispose.mockImplementation(runtime.dispose.bind(runtime))
|
||||||
|
const result = expect(f.start()).resolves.toBe('success')
|
||||||
|
await vi.runAllTimersAsync()
|
||||||
|
await result
|
||||||
|
expect(fetcher).toHaveBeenCalledTimes(2)
|
||||||
|
expect(f.statuses.mock.calls[0]?.[2]).toBe(detail)
|
||||||
|
expect(f.release).toHaveBeenCalledTimes(2)
|
||||||
|
expect(f.dispose).toHaveBeenCalledOnce()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('retries an actual interrupted direct-model stream and retains its reported usage', async () => {
|
||||||
|
let reads = 0
|
||||||
|
const stream = new ReadableStream<Uint8Array>({ pull(controller) {
|
||||||
|
if (reads++ === 0) controller.enqueue(new TextEncoder().encode(
|
||||||
|
'data: {"choices":[{"delta":{"content":"discard me"}}],"usage":{"prompt_tokens":12,"completion_tokens":3}}\n\n'))
|
||||||
|
else controller.error(new TypeError('terminated'))
|
||||||
|
} })
|
||||||
|
const fetcher = vi.fn<typeof fetch>().mockResolvedValueOnce(new Response(stream))
|
||||||
|
.mockResolvedValueOnce(new Response('data: {"choices":[{"delta":{"content":"success"}}],"usage":{"prompt_tokens":14,"completion_tokens":4}}\n\ndata: [DONE]\n\n'))
|
||||||
|
const runtime = new ModelAgentRuntime({ baseUrl: 'https://model.invalid', model: 'fake', authentication: 'none',
|
||||||
|
protocol: 'openai-chat-completions', toolProvider: noModelTools, fetcher })
|
||||||
|
const f = fixture(runtime.run.bind(runtime))
|
||||||
|
f.release.mockImplementation(runtime.releaseConversation.bind(runtime))
|
||||||
|
f.dispose.mockImplementation(runtime.dispose.bind(runtime))
|
||||||
|
const result = expect(f.start()).resolves.toBe('success')
|
||||||
|
await vi.runAllTimersAsync()
|
||||||
|
await result
|
||||||
|
expect(fetcher).toHaveBeenCalledTimes(2)
|
||||||
|
expect(stream.locked).toBe(false)
|
||||||
|
expect(f.db.getTokenUsageSummary().systemTotals).toMatchObject({ callCount: 2, input: 26, output: 7 })
|
||||||
|
expect(f.release).toHaveBeenCalledTimes(2)
|
||||||
|
expect(f.dispose).toHaveBeenCalledOnce()
|
||||||
|
})
|
||||||
@@ -181,6 +181,7 @@ it('stops oversized runtime responses with a readable error while preserving res
|
|||||||
const f = fixture()
|
const f = fixture()
|
||||||
f.respond.mockReturnValue({ ...empty, summary: 'x'.repeat(110_000) })
|
f.respond.mockReturnValue({ ...empty, summary: 'x'.repeat(110_000) })
|
||||||
await expect(f.service(100).run(f.request)).rejects.toThrow('单次模型响应超过 100 KiB')
|
await expect(f.service(100).run(f.request)).rejects.toThrow('单次模型响应超过 100 KiB')
|
||||||
|
expect(f.respond).toHaveBeenCalledOnce()
|
||||||
expect(f.db.listSupervisionResults()).toHaveLength(0)
|
expect(f.db.listSupervisionResults()).toHaveLength(0)
|
||||||
expect(f.db.listSupervisionActivity()[0]!.reviewProgress).toMatchObject({ batches: 0, remainingSources: 1 })
|
expect(f.db.listSupervisionActivity()[0]!.reviewProgress).toMatchObject({ batches: 0, remainingSources: 1 })
|
||||||
const runId = f.db.listSupervisionActivity()[0]!.id
|
const runId = f.db.listSupervisionActivity()[0]!.id
|
||||||
@@ -263,7 +264,7 @@ it('production factory normalizes two absent identities, then reuses exact candi
|
|||||||
expect(f.respond).toHaveBeenCalledTimes(2)
|
expect(f.respond).toHaveBeenCalledTimes(2)
|
||||||
})
|
})
|
||||||
|
|
||||||
it('retains candidate mappings across leaf batches, failed merge and resume without replaying leaves', async () => {
|
it.each([false, true])('retains candidate mappings across leaf batches, failed merge and resume without replaying leaves (transient: %s)', async transient => {
|
||||||
const f = fixture('Atlas and Beacon. '.repeat(100))
|
const f = fixture('Atlas and Beacon. '.repeat(100))
|
||||||
f.db.saveSupervisionResult({ request: f.request, evidence: [], output: { ...empty,
|
f.db.saveSupervisionResult({ request: f.request, evidence: [], output: { ...empty,
|
||||||
entities: ['Atlas', 'Beacon'].map(label => ({ id: label, label, description: label, sourceReferenceIds: [] })) } })
|
entities: ['Atlas', 'Beacon'].map(label => ({ id: label, label, description: label, sourceReferenceIds: [] })) } })
|
||||||
@@ -275,13 +276,15 @@ it('retains candidate mappings across leaf batches, failed merge and resume with
|
|||||||
if (prompt.includes('These inputs are navigation summaries')) {
|
if (prompt.includes('These inputs are navigation summaries')) {
|
||||||
expect(prompt).toContain('KNOWN ENTITIES:\n\n[]')
|
expect(prompt).toContain('KNOWN ENTITIES:\n\n[]')
|
||||||
expect(prompt).toContain('"entities":[]')
|
expect(prompt).toContain('"entities":[]')
|
||||||
if (failMerge) { failMerge = false; throw new Error('Merge interrupted') }
|
if (failMerge) throw transient ? Object.assign(new Error('Merge interrupted'), { status: 503 }) : new Error('Merge interrupted')
|
||||||
return empty
|
return empty
|
||||||
}
|
}
|
||||||
const output = original(prompt) as { entities: Array<{ label: string }> }
|
const output = original(prompt) as { entities: Array<{ label: string }> }
|
||||||
return { ...output, entities: output.entities.map(entity => ({ ...entity, persistedId: candidates.find(candidate => candidate.label === entity.label)!.id })) }
|
return { ...output, entities: output.entities.map(entity => ({ ...entity, persistedId: candidates.find(candidate => candidate.label === entity.label)!.id })) }
|
||||||
})
|
})
|
||||||
await expect(f.service().run(f.request)).rejects.toThrow('Merge interrupted')
|
await expect(f.service().run(f.request)).rejects.toThrow('Merge interrupted')
|
||||||
|
expect(f.respond.mock.calls.filter(([prompt]) => prompt.includes('These inputs are navigation summaries'))).toHaveLength(transient ? 2 : 1)
|
||||||
|
failMerge = false
|
||||||
const runId = f.db.listSupervisionActivity().find(item => item.status === 'failed')!.id
|
const runId = f.db.listSupervisionActivity().find(item => item.status === 'failed')!.id
|
||||||
const saved = f.db.supervisionReviewStore().batches(runId)
|
const saved = f.db.supervisionReviewStore().batches(runId)
|
||||||
expect(saved).toHaveLength(2)
|
expect(saved).toHaveLength(2)
|
||||||
|
|||||||
@@ -20,7 +20,33 @@ type SupervisionModelDependencies = {
|
|||||||
persistUsage: (event: RuntimeModelUsageEvent) => void
|
persistUsage: (event: RuntimeModelUsageEvent) => void
|
||||||
}
|
}
|
||||||
|
|
||||||
/** One bounded, tool-free text call that shares the supervisor concurrency pool. */
|
function isTransientSupervisionError(error: unknown): boolean {
|
||||||
|
let transient = false
|
||||||
|
const seen = new Set<unknown>()
|
||||||
|
for (let current = error; current && typeof current === 'object' && !seen.has(current);) {
|
||||||
|
seen.add(current)
|
||||||
|
const record = current as Record<string, unknown>
|
||||||
|
const message = typeof record.message === 'string' ? record.message : ''
|
||||||
|
const code = typeof record.code === 'string' ? record.code : ''
|
||||||
|
// Direct-model errors retain status; runtime events carry only the public message.
|
||||||
|
const status = record.status ?? record.statusCode ?? /\bHTTP\s+(\d{3})\b/iu.exec(message)?.[1]
|
||||||
|
const failureText = Number(status) === 504 ? message.replace(/\bgateway (?:timeout|timed out)\b/giu, '') : message
|
||||||
|
if (record.name === 'AbortError' || record.name === 'TimeoutError' ||
|
||||||
|
/^(?:ENOTFOUND|ERR_INVALID_URL|CERT_.*|.*CERTIFICATE.*|DEPTH_ZERO_SELF_SIGNED_CERT)$/u.test(code) ||
|
||||||
|
/abort|cancel|timed?\s*out|timeout|超时|取消|容量|context[_ ](?:length|window)|max[_ ]tokens|quota|billing|insufficient|capacity|api[_ -]?key|unauthori[sz]ed|forbidden|authentication|configuration/iu.test(`${code} ${failureText}`)) return false
|
||||||
|
if (status !== undefined) {
|
||||||
|
if (![429, 500, 502, 503, 504].includes(Number(status))) return false
|
||||||
|
transient = true
|
||||||
|
}
|
||||||
|
if (/^(?:ECONNRESET|ECONNREFUSED|EPIPE|EAI_AGAIN|ENETUNREACH|EHOSTUNREACH|UND_ERR_SOCKET)$/u.test(code) ||
|
||||||
|
/^(?:TypeError:\s*)?(?:terminated|fetch failed|network error|socket hang up)$/iu.test(message) ||
|
||||||
|
message === '模型接口流式响应意外中断' || /^rate limit (?:exceeded|reached)\b/iu.test(message)) transient = true
|
||||||
|
current = record.cause
|
||||||
|
}
|
||||||
|
return transient
|
||||||
|
}
|
||||||
|
|
||||||
|
/** One bounded, tool-free text call, with one transient retry in the supervisor pool. */
|
||||||
async function runSupervisionModel(dependencies: SupervisionModelDependencies, request: {
|
async function runSupervisionModel(dependencies: SupervisionModelDependencies, request: {
|
||||||
prompt: string
|
prompt: string
|
||||||
title: string
|
title: string
|
||||||
@@ -36,49 +62,64 @@ async function runSupervisionModel(dependencies: SupervisionModelDependencies, r
|
|||||||
return pool.run(async signal => {
|
return pool.run(async signal => {
|
||||||
const timeoutSeconds = request.timeoutSeconds ?? settings?.supervisorOrganizeTimeoutSeconds ?? defaultSupervisionTimeoutSeconds
|
const timeoutSeconds = request.timeoutSeconds ?? settings?.supervisorOrganizeTimeoutSeconds ?? defaultSupervisionTimeoutSeconds
|
||||||
const runtime = await resolveRuntime()
|
const runtime = await resolveRuntime()
|
||||||
const controller = new AbortController()
|
for (let attempt = 0; ; attempt++) {
|
||||||
const modelSignal = AbortSignal.any([signal, controller.signal, ...(request.signal ? [request.signal] : [])])
|
signal.throwIfAborted()
|
||||||
modelSignal.throwIfAborted()
|
const controller = new AbortController()
|
||||||
let timeoutError: Error | undefined
|
const modelSignal = AbortSignal.any([signal, controller.signal])
|
||||||
const timeout = setTimeout(() => {
|
let timeoutError: Error | undefined
|
||||||
timeoutError = new Error(request.timeoutMessage(timeoutSeconds))
|
const timeout = setTimeout(() => {
|
||||||
controller.abort(timeoutError)
|
timeoutError = new Error(request.timeoutMessage(timeoutSeconds))
|
||||||
}, timeoutSeconds * 1000)
|
controller.abort(timeoutError)
|
||||||
const responseKiB = settings?.supervisionReview?.responseKiB ?? 1024
|
}, timeoutSeconds * 1000)
|
||||||
let output = '', completed = false
|
const responseKiB = settings?.supervisionReview?.responseKiB ?? 1024
|
||||||
const requestId = randomUUID()
|
let output = '', completed = false
|
||||||
const conversationId = `supervision:${requestId}`
|
const requestId = randomUUID()
|
||||||
// Hidden task row so model usage can be attributed to supervision.
|
const conversationId = `supervision:${requestId}`
|
||||||
database.createTask({ id: requestId, conversationId, title: request.title, instructions: request.instructions,
|
|
||||||
origin: 'assistant', visible: false })
|
|
||||||
try {
|
|
||||||
for await (const event of runtime.run({ requestId, conversationId, prompt: request.prompt },
|
|
||||||
modelSignal, async approval => { await request.authorizeTool(approval.toolName ?? approval.scopeKey); return 'deny' })) {
|
|
||||||
if (event.type === 'text') {
|
|
||||||
output += event.delta
|
|
||||||
if (Buffer.byteLength(output) > responseKiB * 1024) {
|
|
||||||
controller.abort(new Error(`单次模型响应超过 ${responseKiB} KiB。请在监督者设置中调高响应容量后继续;已保存批次会保留。`))
|
|
||||||
modelSignal.throwIfAborted()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (event.type === 'model-usage') persistUsage(event)
|
|
||||||
if (event.type === 'tool') throw new Error('监督者只允许只读模型摘要,不允许工具调用')
|
|
||||||
if (event.type === 'error') throw new Error(event.message)
|
|
||||||
if (event.type === 'done') completed = true
|
|
||||||
}
|
|
||||||
modelSignal.throwIfAborted()
|
|
||||||
if (!completed) throw new Error('监督者模型未报告完成')
|
|
||||||
database.updateTaskStatus(requestId, 'completed')
|
|
||||||
return output
|
|
||||||
} catch (error) {
|
|
||||||
const failure = timeoutError ?? error
|
|
||||||
try {
|
try {
|
||||||
database.updateTaskStatus(requestId, modelSignal.aborted ? 'cancelled' : 'failed',
|
// Each attempt has its own hidden task so reported usage survives retries.
|
||||||
failure instanceof Error ? failure.message.slice(0, 2_000) : '监督者模型调用失败')
|
database.createTask({ id: requestId, conversationId, title: request.title, instructions: request.instructions,
|
||||||
} catch { /* status bookkeeping must not mask the model failure */ }
|
origin: 'assistant', visible: false })
|
||||||
throw failure
|
for await (const event of runtime.run({ requestId, conversationId, prompt: request.prompt },
|
||||||
|
modelSignal, async approval => { await request.authorizeTool(approval.toolName ?? approval.scopeKey); return 'deny' })) {
|
||||||
|
if (event.type === 'text') {
|
||||||
|
output += event.delta
|
||||||
|
if (Buffer.byteLength(output) > responseKiB * 1024) {
|
||||||
|
controller.abort(new Error(`单次模型响应超过 ${responseKiB} KiB。请在监督者设置中调高响应容量后继续;已保存批次会保留。`))
|
||||||
|
modelSignal.throwIfAborted()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (event.type === 'model-usage') persistUsage(event)
|
||||||
|
if (event.type === 'tool') throw new Error('监督者只允许只读模型摘要,不允许工具调用')
|
||||||
|
if (event.type === 'error') {
|
||||||
|
const error = new Error(event.message)
|
||||||
|
if (event.status === 'cancelled') controller.abort(error)
|
||||||
|
throw error
|
||||||
|
}
|
||||||
|
if (event.type === 'done') completed = true
|
||||||
|
}
|
||||||
|
modelSignal.throwIfAborted()
|
||||||
|
if (!completed) throw new Error('监督者模型未报告完成')
|
||||||
|
database.updateTaskStatus(requestId, 'completed')
|
||||||
|
return output
|
||||||
|
} catch (error) {
|
||||||
|
const failure = timeoutError ?? (modelSignal.aborted ? modelSignal.reason : error)
|
||||||
|
try {
|
||||||
|
database.updateTaskStatus(requestId, modelSignal.aborted ? 'cancelled' : 'failed',
|
||||||
|
failure instanceof Error ? failure.message.slice(0, 2_000) : '监督者模型调用失败')
|
||||||
|
} catch { /* status bookkeeping must not mask the model failure */ }
|
||||||
|
if (attempt > 0 || modelSignal.aborted || !isTransientSupervisionError(failure)) throw failure
|
||||||
|
} finally {
|
||||||
|
clearTimeout(timeout)
|
||||||
|
controller.abort()
|
||||||
|
await runtime.releaseConversation?.(conversationId)
|
||||||
|
}
|
||||||
|
signal.throwIfAborted()
|
||||||
|
await new Promise<void>((resolve, reject) => {
|
||||||
|
const abort = () => { clearTimeout(wait); reject(signal.reason) }
|
||||||
|
const wait = setTimeout(() => { signal.removeEventListener('abort', abort); resolve() }, 500)
|
||||||
|
signal.addEventListener('abort', abort, { once: true })
|
||||||
|
})
|
||||||
}
|
}
|
||||||
finally { clearTimeout(timeout); await runtime.releaseConversation?.(conversationId) }
|
|
||||||
}, request.signal)
|
}, request.signal)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user