fix: restore remote conversation recovery and clean obsolete agents

Wait for model delivery acknowledgements before completing remote prompts and preserve completed outcomes when idle Runtime cleanup fails. Explicit messages after an unknown outcome now open a fresh binding with history instead of reusing the failed session; recovery remains attach-only.

Run obsolete payload cleanup through the current Agent after activation, retaining history and unresolved process ownership. Live legacy daemons and shared Runtime versions remain deferred. Includes focused regressions, local OpenCode simulation, and isolated Linux validation evidence. Typecheck and lint passed; the full test run timed out.

Release note: Remote conversations can continue after an uncertain result, and completed answers no longer fail solely because Runtime cleanup fails. Updates remove verified inactive Agent payloads and owned orphan processes while retaining active or uncertain installations and conversation history.
This commit is contained in:
mesalogo
2026-09-24 10:39:28 +08:00
parent 0ed1be1773
commit 7adae97baa
22 changed files with 1056 additions and 91 deletions
@@ -161,12 +161,15 @@ daemon 参与。两种 acquisition 的差别只在于如何把同一个签名 co
4. commit 后在同一条已固定身份的 SSH lease 上执行 Agent bootstrap/readiness 和 Runtime
activate。新 Agent 最多完整验证一次,新 Runtime 最多完整验证一次;注册成功后的
health、capabilities 和 prompt 启动只读取 registry、签名 manifest 与入口 metadata。
5. Main 确认 adoption 后显式执行 cleanup。cleanup 失败不回滚已经健康的 Agent/Runtime,
也不阻塞下一次更新。
5. Agent adoption 和所有 Runtime 激活成功后,Main 通过新安装执行 `cleanup-obsolete`,
尽力回收已证明停止且无待处理工作的旧 Agent payload。命令返回逐安装的延期原因,
不支持该命令的旧包或清理失败不会回滚健康环境。随后显式清理本次 operation staging;
staging cleanup 失败也不阻塞下一次更新。
新安装成为 current 后,旧 Agent 排空已有任务再退出,空闲旧 Agent 直接退出;当前 Agent
可以常驻。Runtime 回收、待答问题与旧任务重连的规则见
[远程主机技术设计](./technical-design.md)。
新安装成为 current 后,包含退役逻辑的旧 Agent 排空已有任务再退出;当前 Agent 可以常驻。
仍存活且缺少退役逻辑的 legacy daemon 不会被升级清理命令终止,不能用无连接推断其空闲。
共享 Runtime digest 目录、历史和 Workspace 保留。清理条件、延期限制及验证记录见
[远程主机技术设计](./technical-design.md#升级残留清理)。
这不是两个并行安装管理器,也不存在把 Agent 与 Runtime payload 逐文件走 SFTP 安装的
第二条路径。两种 acquisition 共享相同的 prepare、commit、adoption、finalize 与 cleanup。
+132 -1
View File
@@ -185,7 +185,7 @@ Detached GoodBuddy Agent
解析新的 current registry 并建立当前连接,无需刷新项目记录;当前环境无效时要求用户
显式修复,但不在项目切换中下载或安装。失败或取消保留 Host 配置、凭据、项目、
Workspace 和旧组件。
- 当前 Agent 可以常驻。新安装提升为 current 后,旧 Agent 每 250 ms 检查 registry 并进入
- 当前 Agent 可以常驻。新安装提升为 current 后,已包含退役逻辑的旧 Agent 每 250 ms 检查 registry 并进入
draining,不再接受新 Prompt;无活动任务时退出,有任务时等待完成。已连接 Desktop 的
终态 transcript ACK 也计入排空条件;断线任务完成后不等待离线 Desktop。待答问题仍是
活动任务。旧安装退出前可以重新 Attach 原任务,但不得重新 bootstrap 已退役安装。
@@ -195,6 +195,27 @@ Detached GoodBuddy Agent
- 显式 stop/retire 可校验并停止已被 registry 替换的旧安装。退出只清理 GoodBuddy 自己的
socket、状态和子进程,不删除会话历史,不删除或覆盖无关 Host 文件。
### 升级残留清理
Agent adoption 和所有 Runtime 激活成功后,Main 调用新安装的
`cleanup-obsolete --installation-id <current-id>`。清理由当前 Agent 代码执行,不调用
旧安装的二进制。命令扫描现存 Agent installation 目录,跳过 current 和 candidate,
返回 `complete`、`removed` 与逐安装的 `deferred` 原因。清理不可用或存在延期项时记录
警告,已成功的安装不回滚;下次更新或再次调用该命令时重试,没有后台清理服务。
删除旧 payload 前须完整验证安装包,在既有 bootstrap lock 内确认原 daemon 的生命周期
记录已失效,并只读检查已有 `semantic-prompts.sqlite` 与 journal。活动、结果未知、
未确认的终态、尚存的 ACP channel,以及缺失或无法读取的记录都会阻止清理。随后根据
该安装的 Runtime owner 记录回收可确认归属的孤儿进程;身份冲突、无法检查或仍有子进程
的 owner 记录保留,等待重试。删除前还检查进程是否使用该 payload,并再次核对 registry。
只删除已确认不再使用的 Agent 可执行文件目录,保留 state、历史和 Workspace。
**仍存活的 legacy daemon 不会被此命令回收。** 旧实现没有原子排空并拒绝新工作的操作,
持久记录也不足以证明存活 daemon 的内存任务为空。因此即使没有连接或没有持久操作,
仍返回 `live-or-unproven-daemon-inactivity`,不能把升级后的旧版本自动退出规则追溯应用于
缺少退役逻辑的进程。此轮不裁剪共享 Runtime digest 目录,也不扫描 payload 已丢失而
仅剩 state 的安装。用户报告的六个 daemon、五个旧 helper 和约 2.67 GiB 占用未据此宣告解决。
### Host 级环境生命周期
- Agent、Node 和 Runtime 是 Host/SSH 账号级共享环境。新增或重新验证 Host 后先保存 Host
@@ -909,6 +930,47 @@ goodbuddy-agent diagnostics --installation-id <installationId>
不能以本次 OpenCode 结果替代。历史发布说明中的技术预览记录保留,当前远程项目设置与
说明文档已去除该标签;Linux 桌面控制等其他预览功能不受影响。
## 升级残留清理验证(2026-09-24)
本轮在产品源码改动完成后,重新从工作区构建 cleanup CLI 和隔离验证脚本,并在共享
Linux x64 Host 重跑。使用既有 GoodBuddy 加密凭据存储和固定 Host Key;没有输出凭据。
本机临时 harness 位于 `C:/Users/jiang/AppData/Local/Temp/opencode/`,执行命令为:
```powershell
node "C:\Users\jiang\AppData\Local\Temp\opencode\upgrade-cleanup-build.cjs"
node "C:\Users\jiang\AppData\Local\Temp\opencode\upgrade-cleanup-run.cjs"
```
`upgrade-cleanup-build.cjs` 打包 `upgrade-cleanup-cli.ts`、`upgrade-cleanup-linux.ts`
和 `upgrade-cleanup-host.ts`;run 脚本通过 Electron 启动 host 驱动。Host 驱动读取共享安装
registry 以定位 Node 和 koffi 依赖,仅在 `/root/tmp/gb-upgrade-upload-*` 上传测试脚本。
远端执行形式如下,其中 `<current-id>` 来自共享 registry,`<upload-root>` 是本轮唯一目录:
```sh
NODE_PATH='/root/.goodbuddy/agent/installations/<current-id>/lib/node_modules' TMPDIR='<upload-root>' '<installed-node>' '<upload-root>/linux.cjs' '<upload-root>/cli.cjs'
```
场景在独立 `/root/tmp/gb-upgrade-cleanup-*` HOME 中生成临时签名测试安装和既有格式的
生命周期、语义、journal、owner 记录,通过该 HOME 的 current 安装执行真实 CLI:
`goodbuddy-agent cleanup-obsolete --installation-id current`。测试用实际 Node 进程表示
不含退役逻辑的旧 daemon/helper,不是部署历史发布包或运行真实模型任务。
本次构建与远端运行退出码均为 0。远端断言确认:孤儿 Runtime 被回收,所属旧安装 payload
删除;存活 legacy daemon 延期;另一个带 `starting` 操作记录的进程存活;历史数据库字节
不变;current/candidate payload 保留;第二次调用没有重复删除。运行输出为:
```json
{"passed":true,"productionCli":true,"realOrphanReclaimed":true,"liveLegacyDeferred":true,"activeWorkPreserved":true,"historyPreserved":true,"idempotent":true,"providerCalls":0}
```
测试结束清除自身进程和隔离目录,没有清理共享用户安装或修改其 registry。早先隔离实测
发生在最终补充两个本地回归用例之前;本节记录的是之后重新构建源码的复跑。此前最终本地
9 个定向套件 99 项通过,typecheck 和改动文件 ESLint 通过。
本次三分钟时限内未找到可直接运行 backend/bridge Vitest 套件的 Linux harness,未安装
额外测试依赖,故这两类 Linux 套件未运行。上述 cleanup 场景不替代这些套件,不证明完整
UI 更新或签名复合包安装流程通过;真实模型调用为 0 次。
## 发布前完整回归(不能替代开发验证)
以下清单只用于复核开发期间已经在真实 Host 上通过的链路,不得作为 Agent 改动的首次
@@ -927,3 +989,72 @@ goodbuddy-agent diagnostics --installation-id <installationId>
9. 运行一次有界的真实模型调用,确认凭据不进入 Renderer、SSH 参数、远端环境或磁盘,
只在当前 accepted operation 生命周期内进入 Agent 内存。
10. 中断并恢复 SSH,确认活动 Runtime 不被网络抖动终止。
## Prompt 完成与空闲回收验证(2026-09-24)
Runtime 可以先收到完整模型正文并返回 ACP `end_turn`,随后 helper 才完成 delivery ACK。
`RuntimeAcpBackend` 的正常完成路径现在要求 broker 等待已接收的 exchange 完成,最多等待
2 秒,再关闭 socket;取消等强制关闭路径不等待这一步。ACK 超时或交付失败仍记录
`outcome-unknown`,语义 transcript 与 reconciliation 保持一致。
正常完成时,Agent 在串行 control 操作内先持久化语义终态,再将 binding 标为空闲并回收
Runtime。回收失败记录 `daemon.stop.failed`,reason 为 `idle-runtime-cleanup`,不把已提交的
`completed` 改成未知结果。确定性回归覆盖共享与独占 Runtime 的 stop/reconcile 失败,以及
等待 ACK 期间不得提交终态或停止 Runtime。
Desktop 的下一次显式发送会淘汰缓存中的不可用 binding,再通过 cold-open 创建新请求并
传入已有对话历史。OpenCode 和 Continue 的未知终态均有回归;不会重放原来的未知请求。
`remoteRecoveryOnly` 只允许附加同一活动 operation,已完成或结果未知的缓存不能借恢复
入口发起新请求。缓存仍有活动 operation 时,不同 request ID 的发送被拒绝;普通失败后
已回到 `ready` 的会话仍可继续使用。
### 本机模拟范围
`agent-owned-acp-prompt.local.test.ts` 使用本机已安装的 OpenCode 1.18.29、真实 ACP 和
loopback fake model。测试暂缓 delivery ACK,确认正文 `ACK_OK` 和 ACP 完成先到达,释放
ACK 后提交 `completed`,随后注入清理异常并确认终态不变。该用例注入了
`completePrompt` 回调,没有经过完整 `RuntimeAcpBackend` 完成路径;backend 的提交顺序、
清理诊断和 reconciliation 由 `runtime-acp-backend.test.ts` 的独立回归检查。
本轮本机定向复跑 7 个套件,199 项通过、10 项 Unix-only 用例在 Windows 跳过,耗时
5.60 秒。套件为 `runtime-acp-backend`、`agent-owned-acp-prompt`、
`agent-owned-acp-prompt.local`、`model-bridge`、`model-bridge-client`、
`runtime-composition` 和 `acp-remote-runtime`。本轮没有运行完整测试套件。
### 最终源码 Linux 场景
源码修正完成后,重新构建临时 `lifecycle-host-agent`、`lifecycle-host-daemon` 和
`lifecycle-host-driver`,通过既有 GoodBuddy 加密凭据存储与固定 Host Key 连接共享
Linux x64 Host。执行命令为:
```powershell
node "C:\Users\jiang\AppData\Local\Temp\opencode\lifecycle-host-build.cjs" agent daemon driver
node "C:\Users\jiang\AppData\Local\Temp\opencode\lifecycle-host-run.cjs" driver --fake-model --completion-only
```
测试在独立 `/root/tmp/gb-lifecycle-*` HOME 启动当前源码的 Agent daemon,通过 Desktop
`createManagedRemoteAcpRuntime`、SSH attach、Agent protocol、
`createProductionRuntimeProtocol` 和 `RuntimeAcpBackend` 运行 Host 上已安装的 OpenCode。
该路径使用真实 Runtime process owner、model bridge、ACP 和语义存储,没有注入 backend
的 completion 回调。测试只替换 daemon 的外部模型 HTTP 响应,返回确定的 SSE 文本。
两轮 Ask 均收到完成事件,每轮结束后测试目录所属的 Runtime/helper 进程数为 0。第二轮
在回收后重新启动 Runtime,保留同一原生 session ID,模型请求中带有第一轮保存的标记。
运行输出的关键记录为:
```json
{"inspected":true,"oldState":"ready","draining":false,"remainingOwnedProcesses":0,"realProviderCalls":0,"fakeModelCalls":1}
{"inspected":true,"oldState":"ready","draining":false,"remainingOwnedProcesses":0,"realProviderCalls":0,"fakeModelCalls":2}
{"currentSourceAsk":true,"runtimeRecreated":true,"nativeHistoryPreserved":true}
{"stopped":true,"realProviderCalls":0,"fakeModelCalls":2,"remainingOwnedProcesses":0}
{"exit":0,"stderrBytes":0}
```
构建和运行退出码均为 0,fake model exchange 共 2 次,真实 Provider 请求为 0 次。清理
只停止测试 daemon、其 Runtime 进程并删除隔离目录;没有更改共享用户安装或 registry,
没有停止无关 Host 进程。
此次 Linux 场景验证正常完成、空闲回收和下一轮恢复原生历史;没有在 Host 上注入 ACK
延迟或清理失败,也没有覆盖未知终态后的新 binding 场景、Continue、Execute、断线恢复、
完整 UI 操作或签名包安装。故障时序与未知终态的下一次发送以本机定向回归为证据,不能用
这两轮成功请求替代。
@@ -0,0 +1,107 @@
// @vitest-environment node
import { spawn } from 'node:child_process'
import { existsSync } from 'node:fs'
import { mkdtemp, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { expect, it } from 'vitest'
import { AgentOwnedAcpPrompt } from './agent-owned-acp-prompt'
import { SemanticPromptStore } from './semantic-prompt-store'
import { ModelBridgeLoopbackProxy, createOpenCodeModelBridgeProviderConfig } from './model-bridge-helper'
import type { RuntimeAcpProcessOwner } from './runtime-acp-backend'
const binary = join(process.cwd(), '.runtime-resources', process.arch, process.platform === 'win32' ? 'opencode.exe' : 'opencode')
it.skipIf(!existsSync(binary))('keeps a real OpenCode answer completed after a delayed ACK and cleanup error', async () => {
const root = await mkdtemp(join(tmpdir(), 'goodbuddy-acp-completion-'))
const transcript = new SemanticPromptStore(join(root, 'prompts.sqlite'))
let releaseAck!: () => void
const ack = new Promise<void>(resolve => { releaseAck = resolve })
let enteredCompletion!: () => void
const completion = new Promise<void>(resolve => { enteredCompletion = resolve })
let requests = 0
let acknowledged = false
const body = [
...[{ role: 'assistant', content: 'ACK_OK' }, {}].map((delta, index) => `data: ${JSON.stringify({
id: 'local-ack', object: 'chat.completion.chunk', created: 1, model: 'local-ack',
choices: [{ index: 0, delta, finish_reason: index === 1 ? 'stop' : null }]
})}\n\n`), 'data: [DONE]\n\n'
].join('')
const proxy = new ModelBridgeLoopbackProxy({ exchange: async () => {
if (++requests > 1) throw new Error('Unexpected extra model request')
return {
response: { status: 200, headers: { 'content-type': 'text/event-stream' }, bodyBase64: Buffer.from(body).toString('base64') },
acknowledgeDelivery: async () => { await ack; acknowledged = true },
failDelivery: () => { throw new Error('Unexpected failed delivery') }
}
} })
const origin = await proxy.listen()
const child = spawn(binary, ['acp'], {
cwd: root, stdio: ['pipe', 'pipe', 'pipe'], windowsHide: true,
env: {
PATH: process.env.PATH, SystemRoot: process.env.SystemRoot, HOME: root, USERPROFILE: root,
TEMP: root, TMP: root, XDG_CONFIG_HOME: join(root, 'config'), XDG_DATA_HOME: join(root, 'data'),
XDG_STATE_HOME: join(root, 'state'), XDG_CACHE_HOME: join(root, 'cache'),
OPENCODE_CONFIG_DIR: join(process.cwd(), '.runtime-resources', 'opencode-config'),
OPENCODE_DISABLE_AUTOUPDATE: '1', OPENCODE_DISABLE_MODELS_FETCH: '1', OPENCODE_DISABLE_PROJECT_CONFIG: '1',
OPENCODE_DISABLE_LSP_DOWNLOAD: '1', OPENCODE_DISABLE_EXTERNAL_SKILLS: '1', OPENCODE_DISABLE_CLAUDE_CODE_SKILLS: '1',
OPENCODE_EXPERIMENTAL_DISABLE_FILEWATCHER: '1', OPENCODE_DISABLE_SHARE: '1',
OPENCODE_CONFIG_CONTENT: JSON.stringify(createOpenCodeModelBridgeProviderConfig({
protocol: 'openai-chat-completions', model: 'local-ack', loopbackOrigin: origin, workMode: 'ask'
}))
}
})
child.stderr.resume()
const exited = new Promise<void>(resolve => child.once('close', () => resolve()))
const processOwner = {
writeStdin: (payload: Uint8Array) => new Promise<void>((resolve, reject) => child.stdin.write(payload, error => error ? reject(error) : resolve())),
subscribeOutput: (listener: Parameters<RuntimeAcpProcessOwner['subscribeOutput']>[0]) => {
const onData = (data: Buffer) => { void listener({ stream: 'stdout', data }) }
child.stdout.on('data', onData)
return () => { child.stdout.off('data', onData) }
},
subscribeExit: (listener: () => void) => { child.on('close', listener); return () => { child.off('close', listener) } }
} as RuntimeAcpProcessOwner
const owner = new AgentOwnedAcpPrompt({
bindingId: 'binding', controllerId: 'controller', workspaceDirectory: root, process: processOwner, transcript,
completePrompt: async (_operation, _status, _response, commitTerminal) => {
enteredCompletion()
await ack
commitTerminal()
throw new Error('Simulated post-terminal cleanup failure')
}
})
let timer: NodeJS.Timeout | undefined
try {
transcript.prepare({ bindingId: 'binding', operationId: 'operation', requestId: 'operation', controllerId: 'controller',
preparationDigest: `sha256:${'a'.repeat(64)}`, promptSequence: 0 })
await Promise.race([
(async () => {
await owner.start({ bindingId: 'binding', operationId: 'operation', requestId: 'operation',
prompt: [{ type: 'text', text: 'Reply ACK_OK without using tools.' }] }, 'ask')
await completion
})(),
new Promise<never>((_resolve, reject) => { timer = setTimeout(() => reject(new Error('Local ACP completion timed out')), 20_000) })
])
const page = () => transcript.page({ bindingId: 'binding', operationId: 'operation', controllerId: 'controller', afterSequence: '0', limit: 128 })
expect(acknowledged).toBe(false)
expect(page().state).toBe('running')
expect(JSON.stringify(page().events)).toContain('ACK_OK')
releaseAck()
await expect.poll(() => page().state).toBe('completed')
await expect.poll(() => acknowledged).toBe(true)
expect(page().events.filter(event => event.kind === 'prompt-terminal')).toMatchObject([
{ payload: { status: 'completed', response: { stopReason: 'end_turn' } } }
])
expect(requests).toBe(1)
} finally {
clearTimeout(timer)
releaseAck()
owner.close()
child.kill()
await exited
await proxy.close()
transcript.close()
await rm(root, { recursive: true, force: true })
}
}, 30_000)
@@ -334,7 +334,9 @@ describe('AgentOwnedAcpPrompt', () => {
)
expect(completePrompt).toHaveBeenCalledWith(
'operation-1',
'failed'
'failed',
undefined,
expect.any(Function)
)
expect(
transcript.page({
+25 -48
View File
@@ -36,7 +36,8 @@ export type AgentOwnedAcpPromptOptions = {
completePrompt: (
operationId: string,
status: 'completed' | 'failed' | 'cancelled' | 'outcome-unknown',
response?: PromptResponse
response: PromptResponse | undefined,
commitTerminal: () => void
) => void | Promise<void>
resolveTerminalState?: (
proposed: 'completed' | 'failed' | 'cancelled'
@@ -218,54 +219,30 @@ export class AgentOwnedAcpPrompt {
})
)
this.#promptSettled = promptOperation.then(() => undefined, () => undefined)
const finish = async (proposed: 'completed' | 'failed' | 'cancelled', response?: PromptResponse, error?: unknown) => {
let state = this.#options.resolveTerminalState?.(proposed) ?? proposed
let completionError: unknown
let committed = false
const commitTerminal = () => {
if (committed) return
this.#appendTerminal(state, {
status: state,
...(response === undefined ? { error: boundedError(error) } : { response }),
...(completionError === undefined ? {} : { completionError: boundedError(completionError) })
})
committed = true
}
try {
await this.#options.completePrompt(request.operationId, state, response, commitTerminal)
} catch (error) {
state = 'outcome-unknown'
completionError = error
}
commitTerminal()
}
this.#promptPromise = promptOperation.then(
async (response) => {
const proposed =
response.stopReason === 'cancelled'
? 'cancelled'
: 'completed'
let state =
this.#options.resolveTerminalState?.(proposed) ?? proposed
let completionError: unknown
try {
await this.#options.completePrompt(
request.operationId,
state,
response
)
} catch (error) {
state = 'outcome-unknown'
completionError = error
}
this.#appendTerminal(state, {
status: state,
response,
...(completionError === undefined
? {}
: { completionError: boundedError(completionError) })
})
},
async (error: unknown) => {
let state =
this.#options.resolveTerminalState?.('failed') ?? 'failed'
let completionError: unknown
try {
await this.#options.completePrompt(
request.operationId,
state
)
} catch (completionFailure) {
state = 'outcome-unknown'
completionError = completionFailure
}
this.#appendTerminal(state, {
status: state,
error: boundedError(error),
...(completionError === undefined
? {}
: { completionError: boundedError(completionError) })
})
}
response => finish(response.stopReason === 'cancelled' ? 'cancelled' : 'completed', response),
error => finish('failed', undefined, error)
)
.finally(() => {
this.#questions.clear()
+21 -1
View File
@@ -32,6 +32,7 @@ import {
readManagedAgentReleaseKeyRegistry,
RegisteredAgentBundleError,
verifyPublishedAgentBundle,
verifyInstalledAgentBundle,
type VerifiedInstalledAgentBundle
} from './installed-bundle-verifier'
import { InstallationRegistry } from './installation-registry'
@@ -55,6 +56,7 @@ import {
type DetachedAgentLifecycleOptions
} from './detached-agent-lifecycle'
import { readAgentDiagnostics } from './diagnostic-log'
import { cleanupObsoleteInstallations } from './installation-cleanup'
type CliIo = {
input: Readable
@@ -119,7 +121,8 @@ const LIFECYCLE_ACTIONS = new Set([
'status',
'health',
'stop',
'retire'
'retire',
'cleanup-obsolete'
])
export async function runAgentCli(
@@ -578,6 +581,23 @@ async function runLifecycle(
)
const paths = resolveInstallationPaths(installationId, dependencies)
switch (action) {
case 'cleanup-obsolete': {
const agentRoot = dirname(dirname(dirname(paths.executablePath)))
const releaseKeyRegistry = dependencies.releaseKeyRegistry ??
await readManagedAgentReleaseKeyRegistry(resolve(agentRoot, 'release-keys.json'))
const result = await cleanupObsoleteInstallations({
currentInstallationId: installationId,
agentRoot,
paths: id => resolveInstallationPaths(id, dependencies),
verify: id => (dependencies.verifyInstallation ?? verifyInstalledAgentBundle)(
dirname(resolveInstallationPaths(id, dependencies).executablePath),
{ installationId: id, architecture: currentAgentArchitecture(), releaseKeyRegistry }
),
createLifecycle: dependencies.createLifecycle
})
io.output.write(`${JSON.stringify(result)}\n`)
break
}
case 'adopt': {
const { verified, registry } =
await verifyManagedInstallation(
@@ -39,6 +39,24 @@ afterEach(() => {
})
describe('detached Agent lifecycle', () => {
it('only permits payload cleanup for a stopped recorded daemon under its bootstrap lock', async () => {
const harness = lifecycleHarness()
mkdirSync(harness.stateDirectory, { mode: 0o700 })
const cleanup = vi.fn(async () => {
expect(existsSync(harness.lockPath)).toBe(true)
})
await expect(harness.lifecycle.withStoppedInstallation(cleanup)).resolves.toBe(false)
await harness.becomeReady()
await expect(harness.lifecycle.withStoppedInstallation(cleanup)).resolves.toBe(false)
harness.identities.delete(childPid)
await expect(harness.lifecycle.withStoppedInstallation(cleanup)).resolves.toBe(true)
expect(cleanup).toHaveBeenCalledOnce()
expect(harness.signals).toEqual([])
expect(existsSync(harness.lockPath)).toBe(false)
expect(harness.readRecord()).toBeDefined()
await expect(harness.lifecycle.withStoppedInstallation(async () => { throw new Error('deferred') })).rejects.toThrow('deferred')
expect(existsSync(harness.lockPath)).toBe(false)
})
it('serializes concurrent bootstrap and records endpoint readiness', async () => {
const harness = lifecycleHarness()
harness.onSleep(async () => {
@@ -323,6 +323,21 @@ export class DetachedAgentLifecycle {
return await this.stop()
}
async withStoppedInstallation(action: () => Promise<void>): Promise<boolean> {
const verified = await this.#verifyAndBind()
ensurePrivateDirectory(this.#stateDirectory, { create: false })
const releaseLock = await this.#acquireBootstrapLock()
try {
const status = this.#statusAfterVerification(verified)
// Missing lifecycle evidence is not proof that an older daemon exited.
if (status.state !== 'stale') return false
await action()
return true
} finally {
releaseLock()
}
}
#statusAfterVerification(
verified: VerifiedInstalledAgentBundle
): DetachedLifecycleStatus {
@@ -24,6 +24,27 @@ afterEach(() => {
})
describe('direct Linux stdio Runtime ownership', () => {
it.each(['missing-leader', 'inspection-failed', 'pid-reused'])('retains uncertain orphan ownership for retry (%s)', async scenario => {
const registry = createRegistry()
registry.reserve({ ownerId: 'old', launchId: 'launch-old', processId: 'process-old', installationId: 'installation-old', ownerToken: 'a'.repeat(32) })
registry.markRunning('old', identity())
const sendSignal = vi.fn()
try {
const result = await reconcileOrphanedDirectLinuxStdioProcesses({
installationId: 'installation-old', registry, sendSignal,
readProcessIdentity: async () => {
if (scenario === 'pid-reused') return { ...identity(), startTimeTicks: 999n }
throw Object.assign(new Error('unavailable'), { code: scenario === 'missing-leader' ? 'ENOENT' : 'EACCES' })
},
listPidNamespaceMembers: async () => [{ ...identity(), pid: 43 }]
})
expect(result.unknown + result.conflicts).toBe(1)
expect(registry.get('old')).toBeDefined()
expect(sendSignal).not.toHaveBeenCalled()
} finally {
registry.close()
}
})
it('keeps shared stdin available across Session completion without a cumulative prompt quota', async () => {
const registry = createRegistry()
const child = fakeChild()
@@ -714,7 +714,6 @@ export async function reconcileOrphanedDirectLinuxStdioProcesses(options: {
try {
const actual = await readIdentity(record.processIdentity.pid)
if (!sameLinuxRuntimeProcessIdentity(record.processIdentity, actual)) {
options.registry.remove(record.ownerId, record.state)
conflicts += 1
continue
}
@@ -756,19 +755,14 @@ export async function reconcileOrphanedDirectLinuxStdioProcesses(options: {
} else {
// Without the recorded leader identity there is no safe process
// to match immediately before a group signal.
options.registry.remove(record.ownerId, current.state)
unknown += 1
}
} catch {
const latest = options.registry.get(record.ownerId)
if (latest !== undefined) options.registry.remove(record.ownerId, latest.state)
unknown += 1
}
} else if (error instanceof DirectLinuxStdioProcessOwnerError && error.code === 'identity') {
if (current !== undefined) options.registry.remove(record.ownerId, current.state)
conflicts += 1
} else {
if (current !== undefined) options.registry.remove(record.ownerId, current.state)
unknown += 1
}
}
@@ -0,0 +1,167 @@
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { DatabaseSync } from 'node:sqlite'
import { afterEach, describe, expect, it, vi } from 'vitest'
import { cleanupObsoleteInstallations } from './installation-cleanup'
import { DetachedAgentLifecycle } from './detached-agent-lifecycle'
import { LinuxProcessInspector } from './linux-process-identity'
import { SemanticPromptStore } from './semantic-prompt-store'
import { EventJournal } from './event-journal'
import { RuntimeOwnerRegistry } from './runtime-owner-registry'
import { writePrivateFileAtomic } from './managed-paths'
import type { VerifiedInstalledAgentBundle } from './installed-bundle-verifier'
const roots: string[] = []
afterEach(() => {
vi.restoreAllMocks()
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true })
})
function fixture() {
const root = mkdtempSync(join(tmpdir(), 'goodbuddy-upgrade-cleanup-'))
roots.push(root)
const agentRoot = join(root, 'agent')
const paths = (id: string) => ({
executablePath: join(agentRoot, 'installations', id, 'goodbuddy-agent'),
stateDirectory: join(root, 'state', id),
socketPath: join(root, `${id}.sock`)
})
const verified = (id: string) => ({
installationId: id,
installationDirectory: join(agentRoot, 'installations', id),
executablePath: paths(id).executablePath,
binaryDigest: `sha256:${'a'.repeat(64)}`,
manifest: { protocol: { major: 2, minor: 0 }, entrypoint: { runtimePath: 'node' } }
}) as VerifiedInstalledAgentBundle
for (const id of ['current', 'candidate', 'old', 'older']) {
mkdirSync(join(agentRoot, 'installations', id), { recursive: true, mode: 0o700 })
const state = paths(id).stateDirectory
mkdirSync(join(state, 'journal'), { recursive: true, mode: 0o700 })
writePrivateFileAtomic(paths(id).executablePath, 'payload')
new SemanticPromptStore(join(state, 'semantic-prompts.sqlite')).close()
new EventJournal(join(state, 'journal', 'events.sqlite')).close()
new RuntimeOwnerRegistry(join(state, 'runtime-owners.sqlite')).close()
writePrivateFileAtomic(join(state, 'detached-lifecycle.json'), JSON.stringify({
formatVersion: 1, installationId: id, binaryDigest: verified(id).binaryDigest,
protocol: { major: 2, minor: 0 }, daemonBootId: 'old-boot', createdAt: 1,
process: { pid: 123, starttime: '1', executablePath: join(agentRoot, 'installations', id, 'node') }
}))
}
const entry = (installationId: string) => ({ installationId, agentVersion: '0.11.0', manifestSha256: 'a'.repeat(64), arch: 'x64' })
const registryPath = join(agentRoot, 'registry.json')
writePrivateFileAtomic(registryPath, JSON.stringify({ formatVersion: 1, current: entry('current'), candidate: entry('candidate') }))
const inspector = new LinuxProcessInspector()
vi.spyOn(inspector, 'matches').mockReturnValue(false)
vi.spyOn(inspector, 'inspect').mockReturnValue({ pid: process.pid, starttime: '2', executablePath: process.execPath })
const reconcile = vi.fn(async () => ({ inspected: 0, stopped: 0, conflicts: 0, unknown: 0 }))
const options = {
agentRoot, currentInstallationId: 'current', paths,
verify: vi.fn(async (id: string) => verified(id)),
createLifecycle: (input: ConstructorParameters<typeof DetachedAgentLifecycle>[0]) => new DetachedAgentLifecycle({ ...input, processInspector: inspector }),
reconcile,
payloadInUse: vi.fn(async () => false)
}
return { options, paths, root, registryPath, inspector, reconcile }
}
describe('obsolete installation cleanup', () => {
it('reclaims all proven stopped obsolete payloads, preserves history/current/candidate, and repeats safely', async () => {
const value = fixture()
const history = join(value.paths('old').stateDirectory, 'semantic-prompts.sqlite')
const before = readFileSync(history)
const result = await cleanupObsoleteInstallations(value.options)
expect(result.complete).toBe(true)
expect(result.removed.sort()).toEqual(['old', 'older'])
expect(readFileSync(history)).toEqual(before)
expect(existsSync(value.paths('current').executablePath)).toBe(true)
expect(existsSync(value.paths('candidate').executablePath)).toBe(true)
expect(value.options.verify.mock.calls.flat().sort()).toEqual(['old', 'older'])
await expect(cleanupObsoleteInstallations(value.options)).resolves.toEqual({ complete: true, removed: [], deferred: [] })
})
it('defers live legacy daemons even with no persisted operations or clients', async () => {
const value = fixture()
vi.mocked(value.inspector.matches).mockReturnValue(true)
const result = await cleanupObsoleteInstallations(value.options)
expect(result.removed).toEqual([])
expect(result.deferred).toHaveLength(2)
expect(result.deferred.every(item => item.reason === 'live-or-unproven-daemon-inactivity')).toBe(true)
expect(value.reconcile).not.toHaveBeenCalled()
})
it.each(['starting', 'running', 'outcome-unknown', 'unacknowledged', 'missing', 'raw-channel'])(
'preserves %s work without modifying its authority', async state => {
const value = fixture()
const directory = value.paths('old').stateDirectory
const path = join(directory, 'semantic-prompts.sqlite')
if (state === 'missing') rmSync(path)
else if (state === 'raw-channel') {
const db = new DatabaseSync(join(directory, 'journal', 'events.sqlite'))
db.exec("INSERT INTO acp_channels (binding_id, channel_epoch, direction, controller_id) VALUES ('b', 1, 'main-to-runtime', 'c')")
db.close()
} else {
const store = new SemanticPromptStore(path)
store.prepare({ bindingId: 'b', operationId: 'o', requestId: 'r', controllerId: 'c', preparationDigest: 'd', promptSequence: 0 })
store.close()
const db = new DatabaseSync(path)
db.prepare('UPDATE semantic_prompt_operations SET state = ?, latest_sequence = 1').run(state === 'unacknowledged' ? 'completed' : state)
db.close()
}
const result = await cleanupObsoleteInstallations(value.options)
expect(result.deferred).toContainEqual({ installationId: 'old', reason: 'active-or-unproven-work' })
expect(existsSync(value.paths('old').executablePath)).toBe(true)
if (state !== 'missing' && state !== 'raw-channel') {
const db = new DatabaseSync(path, { readOnly: true })
expect(db.prepare('SELECT state FROM semantic_prompt_operations').get()?.state).toBe(state === 'unacknowledged' ? 'completed' : state)
db.close()
}
}
)
it('retains payloads after uncertain orphan reconciliation and retries later', async () => {
const value = fixture()
value.reconcile.mockResolvedValue({ inspected: 1, stopped: 0, conflicts: 0, unknown: 1 })
const result = await cleanupObsoleteInstallations(value.options)
expect(result.complete).toBe(false)
expect(result.removed).toEqual([])
value.reconcile.mockResolvedValue({ inspected: 0, stopped: 0, conflicts: 0, unknown: 0 })
expect((await cleanupObsoleteInstallations(value.options)).removed).toHaveLength(2)
})
it('does not reinterpret an owner reservation as an empty process tree', async () => {
const value = fixture()
const owners = new RuntimeOwnerRegistry(join(value.paths('old').stateDirectory, 'runtime-owners.sqlite'))
owners.reserve({ ownerId: 'pending', launchId: 'pending', processId: 'pending', installationId: 'old', ownerToken: 'a'.repeat(32) })
owners.close()
const result = await cleanupObsoleteInstallations(value.options)
expect(result.deferred).toContainEqual({ installationId: 'old', reason: 'runtime-ownership-unproven' })
expect(existsSync(value.paths('old').executablePath)).toBe(true)
expect(value.reconcile).toHaveBeenCalledOnce()
})
it('rechecks registry promotion before deleting a payload', async () => {
const value = fixture()
value.options.payloadInUse.mockImplementation(async () => {
const registry = JSON.parse(readFileSync(value.registryPath, 'utf8'))
registry.candidate.installationId = 'old'
writePrivateFileAtomic(value.registryPath, JSON.stringify(registry))
return false
})
const result = await cleanupObsoleteInstallations(value.options)
expect(result.removed).toEqual([])
expect(result.deferred.every(item => item.reason === 'registry-changed')).toBe(true)
expect(existsSync(value.paths('old').executablePath)).toBe(true)
})
it('does not prune when verification, lifecycle evidence, process inspection, or registry identity is uncertain', async () => {
const value = fixture()
value.options.payloadInUse.mockResolvedValue(true)
expect((await cleanupObsoleteInstallations(value.options)).removed).toEqual([])
value.options.payloadInUse.mockResolvedValue(false)
rmSync(join(value.paths('old').stateDirectory, 'detached-lifecycle.json'))
value.options.verify.mockImplementation(async () => { throw new Error('invalid-signature') })
expect((await cleanupObsoleteInstallations(value.options)).removed).toEqual([])
await expect(cleanupObsoleteInstallations({ ...value.options, currentInstallationId: 'older' })).rejects.toThrow('current Agent')
})
})
+146
View File
@@ -0,0 +1,146 @@
import { readdirSync, rmSync, statSync } from 'node:fs'
import { readdir, readlink } from 'node:fs/promises'
import { dirname, join, resolve, sep } from 'node:path'
import { DatabaseSync } from 'node:sqlite'
import { installationRegistryIdSchema, parseInstallationRegistryState } from '../shared/remote-environment-registry-contracts'
import { DetachedAgentLifecycle, type DetachedAgentLifecycleOptions } from './detached-agent-lifecycle'
import { reconcileOrphanedDirectLinuxStdioProcesses } from './direct-linux-stdio-process-owner'
import type { VerifiedInstalledAgentBundle } from './installed-bundle-verifier'
import { assertPrivateRegularFile, ensurePrivateDirectory, readPrivateFile } from './managed-paths'
import { RuntimeOwnerRegistry } from './runtime-owner-registry'
export type InstallationCleanupResult = {
complete: boolean
removed: string[]
deferred: Array<{ installationId: string; reason: string }>
}
export async function cleanupObsoleteInstallations(options: {
currentInstallationId: string
agentRoot: string
paths: (installationId: string) => {
executablePath: string
stateDirectory: string
socketPath: string
}
verify: (installationId: string) => Promise<VerifiedInstalledAgentBundle>
createLifecycle?: (options: DetachedAgentLifecycleOptions) => DetachedAgentLifecycle
reconcile?: typeof reconcileOrphanedDirectLinuxStdioProcesses
payloadInUse?: (directory: string) => Promise<boolean>
}): Promise<InstallationCleanupResult> {
const result: InstallationCleanupResult = { complete: true, removed: [], deferred: [] }
const registryPath = join(options.agentRoot, 'registry.json')
const snapshot = () => parseInstallationRegistryState(JSON.parse(readPrivateFile(registryPath, 1024 * 1024).toString('utf8')))
const initial = snapshot()
if (initial.current?.installationId !== options.currentInstallationId) {
throw new Error('Cleanup must run through the current Agent installation')
}
const installations = join(options.agentRoot, 'installations')
ensurePrivateDirectory(installations, { create: false })
const deadlineAt = new Date(Date.now() + 15_000).toISOString()
for (const entry of readdirSync(installations, { withFileTypes: true })) {
const installationId = entry.name
if (installationId === initial.current.installationId || installationId === initial.candidate?.installationId) continue
const defer = (reason: string) => {
result.complete = false
result.deferred.push({ installationId, reason })
}
if (!installationRegistryIdSchema.safeParse(installationId).success || !entry.isDirectory()) {
defer('unrecognized-installation')
continue
}
let failureReason = 'installation-verification-failed'
try {
if (Date.now() >= Date.parse(deadlineAt)) { defer('cleanup-deadline'); continue }
const paths = options.paths(installationId)
const directory = join(installations, installationId)
if (resolve(dirname(paths.executablePath)) !== resolve(directory)) throw new Error('Installation path mismatch')
const verified = await options.verify(installationId)
const lifecycle = (options.createLifecycle ?? (input => new DetachedAgentLifecycle(input)))({
installationId, ...paths, verifyInstallation: async () => verified
})
const cleaned = await lifecycle.withStoppedInstallation(async () => {
failureReason = 'registry-changed'
if (JSON.stringify(snapshot()) !== JSON.stringify(initial)) throw new Error('Registry changed during cleanup')
// Read the existing authority without constructors that recover or rewrite operations.
failureReason = 'active-or-unproven-work'
assertInactiveInstallation(paths.stateDirectory)
failureReason = 'runtime-ownership-unproven'
const ownersPath = join(paths.stateDirectory, 'runtime-owners.sqlite')
assertPrivateRegularFile(ownersPath)
const database = new DatabaseSync(ownersPath, { readOnly: true })
try {
const table = database.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name IN ('active_runtime_owners', 'runtime_process_owners')").all()
if (table.length !== 1) throw new Error('Unknown Runtime owner schema')
const name = table[0]!.name === 'active_runtime_owners' ? 'active_runtime_owners' : 'runtime_process_owners'
if (database.prepare(`SELECT 1 FROM ${name} WHERE installation_id != ? OR state = 'reserved' OR (state IN ('running', 'stopping') AND process_identity_json IS NULL) LIMIT 1`).get(installationId)) {
throw new Error('Uncertain Runtime ownership')
}
} finally {
database.close()
}
const owners = new RuntimeOwnerRegistry(ownersPath)
try {
failureReason = 'runtime-cleanup-incomplete'
const reconciliation = await (options.reconcile ?? reconcileOrphanedDirectLinuxStdioProcesses)({ installationId, registry: owners, deadlineAt })
if (reconciliation.unknown || reconciliation.conflicts || owners.listForInstallation(installationId).length) {
throw new Error('Runtime cleanup remains uncertain')
}
} finally {
owners.close()
}
failureReason = 'payload-in-use-or-process-inspection-failed'
if (await (options.payloadInUse ?? installationPayloadInUse)(directory)) throw new Error('Installation payload is still in use')
failureReason = 'registry-changed'
if (JSON.stringify(snapshot()) !== JSON.stringify(initial)) throw new Error('Registry changed during cleanup')
// Verification requires an exact executable payload tree. State and history live elsewhere.
failureReason = 'payload-removal-failed'
rmSync(directory, { recursive: true })
result.removed.push(installationId)
})
if (!cleaned) defer('live-or-unproven-daemon-inactivity')
} catch {
defer(failureReason)
}
}
return result
}
function assertInactiveInstallation(stateDirectory: string): void {
ensurePrivateDirectory(stateDirectory, { create: false })
for (const [path, queries] of [
[join(stateDirectory, 'semantic-prompts.sqlite'), [
"SELECT 1 FROM semantic_prompt_operations WHERE state NOT IN ('completed', 'failed', 'cancelled') OR acknowledged_sequence != latest_sequence LIMIT 1"
]],
[join(stateDirectory, 'journal', 'events.sqlite'), [
'SELECT 1 FROM acp_channels LIMIT 1',
'SELECT 1 FROM event_streams WHERE terminal_sequence = 0 OR ack_sequence < last_sequence LIMIT 1'
]]
] as const) {
assertPrivateRegularFile(path)
const database = new DatabaseSync(path, { readOnly: true })
try {
for (const query of queries) if (database.prepare(query).get()) throw new Error('Active or unacknowledged work')
} finally {
database.close()
}
}
}
async function installationPayloadInUse(directory: string): Promise<boolean> {
if (process.platform !== 'linux') return true
for (const entry of await readdir('/proc', { withFileTypes: true })) {
if (!entry.isDirectory() || !/^[1-9]\d*$/u.test(entry.name)) continue
try {
const root = `/proc/${entry.name}`
if (statSync(root).uid !== process.getuid?.()) continue
for (const name of ['exe', 'cwd']) {
const path = await readlink(join(root, name))
if (path === directory || path.startsWith(directory + sep)) return true
}
} catch (error) {
if (!(error instanceof Error && 'code' in error && error.code === 'ENOENT')) throw error
}
}
return false
}
+26 -12
View File
@@ -159,6 +159,7 @@ export class ModelBridgeBrokerServer {
readonly #maximumConnections: number
readonly #requestTimeoutMs: number
readonly #connections = new Set<Socket>()
readonly #requests = new Set<Promise<void>>()
#server?: Server
#socketIdentity?: FileIdentity
#closing = false
@@ -227,7 +228,9 @@ export class ModelBridgeBrokerServer {
socket.once('close', () => {
this.#connections.delete(socket)
})
void this.#serveOne(socket)
const request = this.#serveOne(socket)
this.#requests.add(request)
void request.finally(() => this.#requests.delete(request))
}
)
this.#server = server
@@ -249,12 +252,12 @@ export class ModelBridgeBrokerServer {
}
}
async close(): Promise<void> {
async close(options: { waitForDelivery?: boolean } = {}): Promise<void> {
if (this.#closePromise !== undefined) {
await this.#closePromise
return
}
const closePromise = this.#close()
const closePromise = this.#close(options.waitForDelivery === true)
this.#closePromise = closePromise
try {
await closePromise
@@ -265,20 +268,31 @@ export class ModelBridgeBrokerServer {
}
}
async #close(): Promise<void> {
async #close(waitForDelivery: boolean): Promise<void> {
const server = this.#server
this.#server = undefined
this.#closing = true
try {
for (const connection of this.#connections) {
connection.destroy()
try {
if (waitForDelivery) {
// Runtime can finish its prompt before the helper's delivery ACK arrives.
await raceWithAbort(
Promise.all([...this.#requests]),
AbortSignal.timeout(CLOSE_TIMEOUT_MS)
)
}
} finally {
for (const connection of this.#connections) {
connection.destroy()
}
await settleWithin(
Promise.all([
server === undefined ? Promise.resolve() : closeServer(server),
...this.#requests
]),
CLOSE_TIMEOUT_MS
)
}
await settleWithin(
server === undefined
? Promise.resolve()
: closeServer(server),
CLOSE_TIMEOUT_MS
)
} finally {
this.#connections.clear()
removeSocketIfIdentityMatches(
+52
View File
@@ -615,6 +615,58 @@ describe('model bridge loopback helper', () => {
describe('model bridge Unix broker', () => {
const runOnUnix = process.platform === 'win32' ? it.skip : it
runOnUnix.each(['delivered', 'failed', 'forced', 'timeout'] as const)(
'settles a flushed HTTP response before completion close (%s)', async outcome => {
let releaseAck!: () => void
const ackGate = new Promise<void>(resolve => { releaseAck = resolve })
let enteredAck!: () => void
const ackEntered = new Promise<void>(resolve => { enteredAck = resolve })
const failDelivery = vi.fn()
const broker = new ModelBridgeBrokerServer({
scratchDirectory: privateTemporaryDirectory(),
dispatch: async () => ({
response: validResponse,
acknowledgeDelivery: async () => {
enteredAck()
await ackGate
if (outcome === 'failed') throw new Error('delivery failed')
},
failDelivery
})
})
await broker.listen()
const proxy = new ModelBridgeLoopbackProxy({ exchange: createUnixModelBridgeExchange({ socketPath: broker.socketPath }) })
const origin = await proxy.listen()
try {
const response = await sendHttp(origin, { path: '/v1/messages', body: '{}' })
expect(response.body).toBe('{"ok":true}')
await ackEntered
let closed = false
const close = broker.close({ waitForDelivery: outcome !== 'forced' }).then(() => { closed = true })
// An event-loop turn proves completion close has not torn down the ACK socket.
await new Promise<void>(resolve => setImmediate(resolve))
if (outcome === 'forced') {
await close
expect(failDelivery).toHaveBeenCalledOnce()
} else if (outcome === 'timeout') {
await expect(close).rejects.toThrow()
expect(failDelivery).toHaveBeenCalledOnce()
} else {
expect(closed).toBe(false)
expect(failDelivery).not.toHaveBeenCalled()
releaseAck()
await close
expect(failDelivery).toHaveBeenCalledTimes(outcome === 'failed' ? 1 : 0)
}
expect(existsSync(broker.socketPath)).toBe(false)
} finally {
releaseAck()
await proxy.close()
await broker.close()
}
}
)
runOnUnix('routes shared Sessions concurrently and rejects queued work after its operation ends', async () => {
let unblock!: () => void
const blocked = new Promise<void>(resolve => { unblock = resolve })
@@ -283,6 +283,64 @@ class FakeProcess implements RuntimeAcpProcessOwner {
type Harness = ReturnType<typeof harness>
function deferred<T = void>() {
let resolve!: (value: T) => void
let reject!: (error: unknown) => void
const promise = new Promise<T>((yes, no) => { resolve = yes; reject = no })
return { promise, resolve, reject }
}
async function ownedHarness(shared: boolean) {
const root = await mkdtemp(join(tmpdir(), 'goodbuddy-terminal-race-'))
const transcript = new SemanticPromptStore(join(root, 'prompts.sqlite'))
const prompts: Array<() => Promise<void>> = []
const processes: FakeProcess[] = []
const setRoute = vi.spyOn(AgentAcpConnection.prototype, 'setModelRoute').mockResolvedValue()
const fixture = harness({
agentOwned: true, shareOwnedProcesses: shared, semanticStore: transcript, uniqueProcesses: true,
configureProcess: process => {
processes.push(process)
process.onWrite = payload => {
const request = JSON.parse(Buffer.from(payload).toString())
if (request.id === undefined) return
const reply = (result: unknown) => process.emit('stdout', `${JSON.stringify({ jsonrpc: '2.0', id: request.id, result })}\n`)
if (request.method === 'session/prompt') {
prompts.push(() => reply({ stopReason: 'end_turn' }))
return
}
let result: unknown = {}
if (request.method === 'initialize') result = { protocolVersion: 1, agentCapabilities: {} }
if (request.method === 'session/new') result = { sessionId: `session-${processes.length}` }
if (request.method === 'session/set_config_option') result = {
configOptions: [{ id: 'model', name: 'Model', type: 'select', currentValue: request.params.value, options: [] }]
}
void reply(result)
}
}
})
await open(fixture)
return {
...fixture, transcript, prompts, processes,
start: async (operationId = 'request-1', deadlineAt = UNBOUNDED_REMOTE_PROMPT_DEADLINE) => {
await invoke(fixture, 'runtime/preparePrompt', fixture.preparation({ operationId, requestId: operationId, deadlineAt }))
await invoke(fixture, 'runtime/startPrompt', {
bindingId: 'binding-1', operationId, requestId: operationId, prompt: [{ type: 'text', text: 'Finish' }]
})
},
page: (operationId = 'request-1') => transcript.page({
bindingId: 'binding-1', operationId, controllerId: fixture.context.controller.controllerId,
afterSequence: '0', limit: 10
}),
cleanup: async () => {
try { await fixture.backend.dispose() } finally {
setRoute.mockRestore()
transcript.close()
await rm(root, { recursive: true, force: true })
}
}
}
}
function harness(input: {
workMode?: 'ask' | 'execute'
now?: number
@@ -465,6 +523,74 @@ function harness(input: {
}
describe('RuntimeAcpBackend', () => {
it.each([false, true])('commits completion before idle cleanup and preserves it on cleanup failure (shared=%s)', async shared => {
for (const failure of ['stop', 'reconcile'] as const) {
const fixture = await ownedHarness(shared)
const enteredStop = deferred()
const releaseStop = deferred()
const stop = vi.spyOn(fixture.process, 'stop').mockImplementationOnce(async () => {
enteredStop.resolve()
await releaseStop.promise
if (failure === 'stop') throw new Error('idle stop failed')
})
try {
await fixture.start()
await fixture.prompts[0]!()
await enteredStop.promise
expect(fixture.page()).toMatchObject({ state: 'completed', events: [{ kind: 'prompt-terminal', payload: { status: 'completed' } }] })
releaseStop.resolve()
await expect(invoke(fixture, 'runtime/reconcilePrompt', {
bindingId: 'binding-1', operationId: 'request-1', requestId: 'request-1'
})).resolves.toMatchObject({ status: 'terminal', terminalState: 'completed' })
expect(fixture.diagnostics.tryRecord).toHaveBeenCalledWith('daemon.stop.failed', expect.objectContaining({
reason: 'idle-runtime-cleanup', error: expect.any(Error)
}))
expect(fixture.page().events).toHaveLength(1)
expect(fixture.page().state).toBe('completed')
} finally {
releaseStop.resolve()
stop.mockRestore()
await fixture.cleanup()
}
}
})
it('waits for bridge delivery before committing the owned terminal or stopping Runtime', async () => {
const fixture = await ownedHarness(true)
const enteredClose = deferred()
const delivered = deferred()
fixture.bridgeCloses.mockImplementationOnce(async () => {
enteredClose.resolve()
await delivered.promise
})
try {
await fixture.start()
await fixture.prompts[0]!()
await enteredClose.promise
expect(fixture.bridgeCloses).toHaveBeenCalledWith({ waitForDelivery: true })
expect(fixture.page().state).toBe('running')
expect(fixture.process.stops).toEqual([])
delivered.resolve()
await invoke(fixture, 'runtime/getAcpCursors', { bindingId: 'binding-1' })
expect(fixture.page().state).toBe('completed')
expect(fixture.process.stops).toEqual(['binding-closed'])
} finally { delivered.resolve(); await fixture.cleanup() }
})
it.each([false, true])('keeps transcript and reconciliation unknown when completion delivery fails (shared=%s)', async shared => {
const fixture = await ownedHarness(shared)
fixture.bridgeCloses.mockRejectedValueOnce(new Error('delivery ACK timed out'))
try {
await fixture.start()
await fixture.prompts[0]!()
await vi.waitFor(() => expect(fixture.page().state).toBe('outcome-unknown'))
await expect(invoke(fixture, 'runtime/reconcilePrompt', {
bindingId: 'binding-1', operationId: 'request-1', requestId: 'request-1'
})).resolves.toMatchObject({ status: 'outcome-unknown' })
expect(fixture.page().events).toHaveLength(1)
} finally { await fixture.cleanup() }
})
it('binds MCP image calls to authenticated blob framing and closes each prompt endpoint', async () => {
const fixture = harness({ workMode: 'execute' })
const client = new Client({ name: 'image-backend-test', version: '1' })
+29 -16
View File
@@ -304,7 +304,7 @@ type PreparedOperation = {
acceptance: RemotePromptOperationAcceptance
budget: RemotePromptOperationPreparation['budget']
completion?: z.infer<typeof acpCompletePromptResultSchema>
terminalState?: 'completed' | 'failed' | 'cancelled' | 'interrupted'
terminalState?: 'completed' | 'failed' | 'cancelled' | 'interrupted' | 'outcome-unknown'
processTree?: 'running' | 'empty'
modelProfile?: AgentPromptModelProfile
promptSequence: number
@@ -1529,12 +1529,13 @@ export class RuntimeAcpBackend {
}
} : {}),
transcript: this.#options.semanticPrompts,
completePrompt: async (operationId, status) => {
completePrompt: async (operationId, status, _response, commitTerminal) => {
await this.#enqueueControl(async () => {
await this.#terminalizeOwnedPrompt(
binding,
operationId,
status
status,
commitTerminal
)
})
},
@@ -1653,7 +1654,8 @@ export class RuntimeAcpBackend {
async #terminalizeOwnedPrompt(
binding: BindingState,
operationId: string,
status: 'completed' | 'failed' | 'cancelled' | 'outcome-unknown'
status: 'completed' | 'failed' | 'cancelled' | 'outcome-unknown',
commitTerminal: () => void
): Promise<void> {
const prepared = binding.operations.get(operationId)
if (
@@ -1663,16 +1665,15 @@ export class RuntimeAcpBackend {
) {
return
}
await this.#closeModelBridge(binding, true)
try {
await this.#closeModelBridge(binding, true, false, true)
if (!binding.sharedProcess) await binding.process.completePrompt()
} catch {
} catch (error) {
binding.poisoned = true
binding.state = 'outcome-unknown'
prepared.modelProfile = undefined
throw new RuntimeAcpBackendError(
'Runtime prompt completion failed',
'process'
)
await this.#stopAndReconcile(binding, 'identity-conflict').catch(() => undefined)
throw error
}
if (binding.poisoned || status === 'outcome-unknown') {
binding.state = 'outcome-unknown'
@@ -1683,6 +1684,8 @@ export class RuntimeAcpBackend {
).catch(() => undefined)
return
}
// Persist the terminal while control is serialized, before exposing an idle binding.
commitTerminal()
prepared.terminalState =
status === 'cancelled'
? 'cancelled'
@@ -1710,7 +1713,15 @@ export class RuntimeAcpBackend {
clearTimeout(binding.ownedPromptStartTimer)
binding.ownedPromptStartTimer = undefined
}
await this.#releaseIdleProcess(binding)
try {
await this.#releaseIdleProcess(binding)
} catch (error) {
this.#options.diagnostics?.tryRecord('daemon.stop.failed', {
runtimeId: binding.request.runtimeId,
reason: 'idle-runtime-cleanup',
error
})
}
}
async #startModelBridge(
@@ -1887,7 +1898,8 @@ export class RuntimeAcpBackend {
async #closeModelBridge(
binding: BindingState,
poisonIfActive: boolean,
preserveRuntime = false
preserveRuntime = false,
waitForDelivery = false
): Promise<void> {
binding.imageTool?.close()
binding.imageTool = undefined
@@ -1902,7 +1914,7 @@ export class RuntimeAcpBackend {
).catch(() => undefined)
}
try {
await broker?.close()
await broker?.close({ waitForDelivery })
} catch (error) {
brokerError = error
}
@@ -2039,6 +2051,9 @@ export class RuntimeAcpBackend {
processTree: prepared.completion.processTree
}
}
if (prepared.terminalState === 'outcome-unknown') {
return { status: 'outcome-unknown', processTree: 'unknown' }
}
if (prepared.terminalState !== undefined) {
return {
status: 'terminal',
@@ -2445,9 +2460,7 @@ export class RuntimeAcpBackend {
},
terminalState: terminal.status
})
if (terminal.status !== 'outcome-unknown') {
prepared.terminalState = terminal.status
}
prepared.terminalState = terminal.status
} catch {
// A concurrent Agent-owned terminal transition already won.
}
+98
View File
@@ -3494,6 +3494,104 @@ describe('AcpRemoteRuntime Agent-owned prompts', () => {
}
})
it.each(['opencode', 'continue'] as const)('starts a fresh %s binding after a cached unknown terminal', async runtimeId => {
const fixture = ownedChannel({ runtimeId, state: 'outcome-unknown' })
const next = ownedChannel({ runtimeId })
try {
await expect(collect(fixture.instance.run(request, new AbortController().signal)))
.rejects.toThrow(/终态未知/iu)
const original = (await fixture.store.getByConversation(request.conversationId))!
expect(original).toMatchObject({ state: 'outcome-unknown', activePromptOperationId: request.requestId })
await expect(collect(fixture.instance.run(
{ ...request, remoteRecoveryOnly: true }, new AbortController().signal
))).rejects.toBeInstanceOf(RemotePromptRecoveryUnavailableError)
expect(fixture.channel.close).not.toHaveBeenCalled()
fixture.pageOwnedPromptTranscript.mockImplementation(next.pageOwnedPromptTranscript)
const events = await collect(fixture.instance.run({
...request, requestId: 'after-unknown', prompt: 'new explicit request',
history: [{ role: 'user', content: 'previous question' }, { role: 'assistant', content: 'previous answer' }]
}, new AbortController().signal))
expect(events.some(event => event.type === 'done')).toBe(true)
expect(await fixture.store.getById(original.bindingId)).toMatchObject({ state: 'closed', activePromptOperationId: undefined })
const binding = (await fixture.store.getByConversation(request.conversationId))!
expect(binding.bindingId).not.toBe(original.bindingId)
expect(binding).toMatchObject({ state: 'ready', promptSequence: 0 })
expect(fixture.channel.close).toHaveBeenCalledOnce()
expect(fixture.preparePrompt).toHaveBeenCalledTimes(2)
expect(fixture.startOwnedPrompt).toHaveBeenCalledTimes(2)
expect(fixture.startOwnedPrompt.mock.calls[1]![0]).toMatchObject({
bindingId: binding.bindingId, operationId: 'after-unknown', requestId: 'after-unknown'
})
expect(fixture.startOwnedPrompt.mock.calls[1]![0]).not.toHaveProperty('acpSessionId')
expect(JSON.stringify(fixture.startOwnedPrompt.mock.calls[1]![0])).toContain('previous answer')
expect(JSON.stringify(fixture.startOwnedPrompt.mock.calls[1]![0])).toContain('new explicit request')
expect(fixture.attachOwnedPrompt).not.toHaveBeenCalled()
} finally { await fixture.instance.dispose(); await next.instance.dispose() }
})
it.each(['opencode', 'continue'] as const)('does not prepare or start attach-only recovery from a cached ready %s session', async runtimeId => {
const fixture = ownedChannel({ runtimeId })
try {
await collect(fixture.instance.run(request, new AbortController().signal))
const binding = await fixture.store.getByConversation(request.conversationId)
await expect(collect(fixture.instance.run(
{ ...request, remoteRecoveryOnly: true }, new AbortController().signal
))).rejects.toBeInstanceOf(RemotePromptRecoveryUnavailableError)
expect(await fixture.store.getByConversation(request.conversationId)).toEqual(binding)
expect(fixture.preparePrompt).toHaveBeenCalledOnce()
expect(fixture.startOwnedPrompt).toHaveBeenCalledOnce()
expect(fixture.attachOwnedPrompt).not.toHaveBeenCalled()
expect(fixture.channel.close).not.toHaveBeenCalled()
} finally { await fixture.instance.dispose() }
})
it('preserves a cached running operation when another explicit request arrives', async () => {
const fixture = ownedChannel()
try {
const stream = fixture.instance.run(request, new AbortController().signal)
await stream.next()
await stream.return()
const binding = await fixture.store.getByConversation(request.conversationId)
await expect(collect(fixture.instance.run(
{ ...request, requestId: 'different-request' }, new AbortController().signal
))).rejects.toThrow(/必须先恢复或取消原请求/iu)
expect(await fixture.store.getByConversation(request.conversationId)).toEqual(binding)
expect(fixture.preparePrompt).toHaveBeenCalledOnce()
expect(fixture.startOwnedPrompt).toHaveBeenCalledOnce()
expect(fixture.channel.close).not.toHaveBeenCalled()
const recovered = await collect(fixture.instance.run(
{ ...request, remoteRecoveryOnly: true }, new AbortController().signal
))
expect(recovered.some(event => event.type === 'done')).toBe(true)
expect(fixture.preparePrompt).toHaveBeenCalledOnce()
expect(fixture.startOwnedPrompt).toHaveBeenCalledOnce()
} finally { await fixture.instance.dispose() }
})
it('continues a cached ready session after a failed terminal', async () => {
const fixture = ownedChannel({ state: 'failed' })
const next = ownedChannel()
try {
await expect(collect(fixture.instance.run(request, new AbortController().signal)))
.rejects.toThrow('owned failure')
const binding = (await fixture.store.getByConversation(request.conversationId))!
expect(binding).toMatchObject({ state: 'ready', activePromptOperationId: undefined })
fixture.pageOwnedPromptTranscript.mockImplementation(next.pageOwnedPromptTranscript)
const events = await collect(fixture.instance.run(
{ ...request, requestId: 'after-failure' }, new AbortController().signal
))
expect(events.some(event => event.type === 'done')).toBe(true)
expect(await fixture.store.getByConversation(request.conversationId)).toMatchObject({
bindingId: binding.bindingId, state: 'ready', promptSequence: binding.promptSequence + 1
})
expect(fixture.startOwnedPrompt.mock.calls[1]![0]).toHaveProperty('acpSessionId', binding.acpSessionId)
expect(fixture.preparePrompt).toHaveBeenCalledTimes(2)
expect(fixture.startOwnedPrompt).toHaveBeenCalledTimes(2)
expect(fixture.attachOwnedPrompt).not.toHaveBeenCalled()
expect(fixture.channel.close).not.toHaveBeenCalled()
} finally { await fixture.instance.dispose(); await next.instance.dispose() }
})
it('never starts a replacement prompt for attach-only recovery', async () => {
const fixture = ownedChannel()
const recoveryRequest = {
+25
View File
@@ -1746,6 +1746,31 @@ export class AcpRemoteRuntime implements AgentRuntime {
signal: AbortSignal
): Promise<SessionRecord> {
let current = this.sessions.get(conversationId)
if (current && request.remoteRecoveryOnly === true && !(
current.binding.state === 'prompt-running' &&
current.binding.activePromptOperationId === request.requestId
)) {
throw new RemotePromptRecoveryUnavailableError(
'没有可安全附加的远端 Agent 请求,且恢复不会重放任务'
)
}
if (current?.binding.state === 'prompt-running' &&
this.options.modelProfile !== undefined &&
current.binding.activePromptOperationId !== request.requestId) {
throw new Error('当前对话仍有 Agent 托管请求,必须先恢复或取消原请求')
}
if (current && current.binding.state !== 'ready' &&
current.binding.state !== 'prompt-running') {
// Let cold-open retire an unusable binding before starting a new request.
this.sessions.delete(conversationId)
this.contexts.delete(conversationId)
await this.awaitLocalOperation(
'关闭结果未知会话通道',
this.closeContext(current.context),
signal
).catch(() => undefined)
current = undefined
}
if (current?.binding.state === 'ready' && this.options.runtimeId === 'continue' &&
this.options.modelProfile !== undefined) {
// Continue keeps native history in process memory; seed the next process
@@ -102,6 +102,7 @@ type FixtureOptions = {
uploadFile?: ReturnType<typeof vi.fn>
activateAgent?: ReturnType<typeof vi.fn>
activateRuntime?: ReturnType<typeof vi.fn>
cleanupObsolete?: () => Promise<{ exitCode: number; stdout: string; stderr: string }>
pendingOperation?: PendingRemoteEnvironmentOperation
}
@@ -165,7 +166,12 @@ function fixture(options: FixtureOptions = {}) {
runAgentLifecycleAction: vi.fn(async (
installationId: string,
action: string
) => (activateAgent as (
) => action === 'cleanup-obsolete'
? (options.cleanupObsolete ?? (async () => {
calls.push('cleanup-obsolete')
return { exitCode: 0, stdout: '{"complete":true}', stderr: '' }
}))()
: (activateAgent as (
installationId: string,
action: string
) => Promise<unknown>)(installationId, action)),
@@ -327,6 +333,19 @@ function fixture(options: FixtureOptions = {}) {
}
describe('RemoteEnvironmentPreparer', () => {
it.each([0, 2])('reports deferred or unsupported cleanup without failing a healthy update (exit=%s)', async exitCode => {
const warning = vi.spyOn(console, 'warn').mockImplementation(() => {})
const cleanupObsolete = vi.fn(async () => ({ exitCode, stdout: '{"complete":false}', stderr: '' }))
const value = fixture({ cleanupObsolete })
try {
await expect(value.preparer.prepare(HOST_ID, 'remote-download', undefined, new AbortController().signal)).resolves.toBeUndefined()
expect(cleanupObsolete).toHaveBeenCalledOnce()
expect(value.regularLease.runAgentLifecycleAction).toHaveBeenLastCalledWith(identity.agent.installationId, 'cleanup-obsolete', expect.any(AbortSignal))
expect(warning).toHaveBeenCalledWith(expect.stringContaining('incomplete'))
} finally {
warning.mockRestore()
}
})
it('runs the remote compound graph in order, verifies identity, cleans explicitly, and emits the resolved method', async () => {
const value = fixture()
const progress: Array<{ method: string; phase: string }> = []
@@ -352,6 +371,7 @@ describe('RemoteEnvironmentPreparer', () => {
'commit',
'agent',
'runtime',
'cleanup-obsolete',
'cleanup',
'remove-operation',
'release-bootstrap',
@@ -406,6 +406,16 @@ export class RemoteEnvironmentPreparer implements Preparer {
)
signal.throwIfAborted()
}
try {
const cleanup = await lease.runAgentLifecycleAction(installationId, 'cleanup-obsolete', signal)
if (cleanup.exitCode !== 0 || JSON.parse(cleanup.stdout).complete !== true) {
console.warn('Obsolete Agent cleanup is incomplete; active or unproven installations were retained')
}
} catch {
// Cleanup does not invalidate an adopted environment, including older packages without this action.
console.warn('Obsolete Agent cleanup is unavailable; installed payloads were retained')
}
signal.throwIfAborted()
}
async #prepareUploadedArchive(options: {
+5
View File
@@ -29,6 +29,11 @@ function bootstrapOutput(
}
describe('fixed Agent SSH commands', () => {
it('runs obsolete cleanup through the current installation, never an obsolete binary', () => {
expect(buildFixedAgentSshCommand(verifyAgentInstallationId('agent-current'), {
kind: 'lifecycle', action: 'cleanup-obsolete'
})).toBe('exec "$HOME/.goodbuddy/agent/installations/agent-current/goodbuddy-agent" cleanup-obsolete --installation-id agent-current')
})
it('uses one exact, fixed, non-mutating bootstrap probe command', () => {
expect(AGENT_BOOTSTRAP_PROBE_COMMAND).toBe(
`printf 'GOODBUDDY_AGENT_BOOTSTRAP_PROBE_V1\\n'; ` +
+1
View File
@@ -29,6 +29,7 @@ const lifecycleActions = [
'health',
'status',
'retire',
'cleanup-obsolete',
'stop'
] as const
const runtimeArchitectures = ['x64', 'arm64'] as const