# Agent Instructions - ZStack AIOS (GPUStack downstream fork)

This is the **combined workspace** for our ZStack AIOS product, a downstream fork of
[GPUStack](https://github.com/gpustack). It bundles both the frontend and backend
repos so agents and contributors work from a single root going forward.

## Layout

```
gpustack-all-lofyer/          <- this repo (tooling only)
├── AGENTS.md                 # workspace-level conventions
├── scripts/
│   ├── bootstrap             # clone the two forks into place
│   ├── build-image           # frontend build + image build + verify
│   ├── verify-image          # smoke test a built image
│   ├── sync-upstream         # bump both forks to a new upstream release
│   └── lib/common.sh         # shared helpers
├── gpustack-ui-lofyer/       # frontend fork  (upstream: gpustack/gpustack-ui)
└── gpustack-lofyer/          # backend  fork  (upstream: gpustack/gpustack)
```

This repo tracks **only the workspace tooling**. The two forks live in their own
repositories and are *not* submodules, so each keeps a clean upstream history that
can be rebased onto new releases. After cloning this repo, run `scripts/bootstrap`
to lay the forks out at the paths above.

Each sub-repo is an independent git repository with its own history, remotes, and
`AGENTS.md` / `CLAUDE.md`. Read the sub-repo's own instructions before working in it.

Remotes (`origin` = private registry, `upstream` = GitHub):

| Repo               | origin                                              |
| ------------------ | --------------------------------------------------- |
| gpustack-all-lofyer| `ssh://git@git.digiman.live:11022/root/gpustack-all-lofyer.git` |
| gpustack-ui-lofyer | `ssh://git@git.digiman.live:11022/root/gpustack-ui.git`         |
| gpustack-lofyer    | `ssh://git@git.digiman.live:11022/root/gpustack.git`            |

## Workspace scripts

Run these from the workspace root. Every script refuses to proceed unless both
forks sit on the **same** `v<version>-lofyer` branch, so a build can never mix a
frontend and backend from different upstream releases.

```bash
scripts/bootstrap [v<version>]   # clone both forks, checkout v<version>-lofyer
scripts/build-image              # npm build + image build + full verification
scripts/build-image --skip-ui-build   # reuse an existing frontend dist/
scripts/build-image --no-verify       # build only
scripts/build-image --push            # push the image after a green build
scripts/build-image --dry-run         # show the plan
scripts/verify-image             # smoke test the current version's image
scripts/verify-image --static-only    # skip booting a container
scripts/sync-upstream --list     # upstream releases available in BOTH repos
scripts/sync-upstream v2.2.3     # bump both forks to v2.2.3
scripts/sync-upstream v2.2.3 --dry-run
scripts/sync-upstream --continue # resume after resolving a cherry-pick conflict
```

## Versioning

Both repos track the **same upstream release** and keep customizations on a
`v<upstream-version>-lofyer` branch. Frontend and backend are always bumped
**together** to the same upstream tag. Current target: **v2.2.3**.

| Repo               | Branch          | Upstream base | Upstream repo           |
| ------------------ | --------------- | ------------- | ----------------------- |
| gpustack-ui-lofyer | `v2.2.3-lofyer` | tag `v2.2.3`  | `gpustack/gpustack-ui`  |
| gpustack-lofyer    | `v2.2.3-lofyer` | tag `v2.2.3`  | `gpustack/gpustack`     |

## Branch rules

- **Never commit customizations to `main` or to any upstream branch.** `main`
  mirrors upstream and must stay clean so it can be fast-forwarded/mirrored.
- All customization work lives on `v<upstream-version>-lofyer` (e.g.
  `v2.2.1-lofyer`), one such branch per upstream release we follow.
- One upstream version = one `-lofyer` branch. Do **not** reuse an old branch for a
  new upstream version; cut a fresh one from the new tag and re-apply commits.
- Keep the frontend and backend `-lofyer` branch names in lockstep (same version
  suffix) so a customer build always pairs matching branches.
- Only fast-forward-safe, curated customization commits belong on `-lofyer`. Prefer
  small, cherry-pickable commits so the next version bump replays cleanly.

## Updating to a new upstream version

Use `scripts/sync-upstream`, which does the whole bump for both repos:

```bash
scripts/sync-upstream --list            # releases available in BOTH upstreams
scripts/sync-upstream v2.2.3 --dry-run  # review the replay plan first
scripts/sync-upstream v2.2.3            # cut branches + replay fork commits
scripts/build-image                     # verify the result end to end
git -C gpustack-ui-lofyer push -u origin v2.2.3-lofyer
git -C gpustack-lofyer   push -u origin v2.2.3-lofyer
# then update the version table + "Current target" above
```

What it does per repo: fetch `upstream`, require a clean worktree and that the tag
exists in **both** upstreams, cut `v<version>-lofyer` from the tag, then cherry-pick
our fork-only commits oldest-first. On conflict it stops and tells you exactly what
to resolve; afterwards run `scripts/sync-upstream --continue`, then re-run with the
target version to replay the remainder. Nothing is pushed automatically.

**Identifying fork-only commits** is the subtle part, and the script handles it:
`git log --cherry-pick --right-only` alone is *not* sufficient, because upstream
commits that were rebased or squashed into the new release get a different patch-id
and would be replayed a second time. The script therefore also drops any commit that
an `upstream/*` branch or any tag contains. If you ever do this by hand, verify with
`git branch -r --contains <sha> | grep upstream`.

Untracked files (e.g. work in progress under `src/pages/`) are treated as yours and
left untouched; only tracked modifications block the bump.

## Customizations to re-apply each version

### Frontend (`gpustack-ui-lofyer`)

Fork-only commits carried on top of upstream. Details in
`gpustack-ui-lofyer/AGENTS.md` + `CLAUDE.md`. Notably:

- `scripts/sync-github` and `scripts/rebrand` (the tooling itself).
- Rebrand user-facing `GPUStack` -> `ZStack AIOS` (run `scripts/rebrand`).
- Footer/topbar trim, navigation moved to top header.

### Backend (`gpustack-lofyer`)

- **API key / token prefix**: `API_KEY_PREFIX` in `gpustack/security.py` is set to
  `zstack` (upstream `gpustack`). This drives API keys, worker registration
  tokens, cluster registration tokens (`zstack_{access}_{secret}`), and the
  masked display value. We intentionally **do not** stay backward compatible with
  old `gpustack_` keys (fresh customer deployments only). Keep the assertions in
  `tests/utils/test_api_keys.py` in sync with the prefix.
- **Image build tooling**: `pack/Dockerfile.lofyer` + `hack/package-lofyer` (thin
  overlay build, see "Shipping a customer build" below). These are fork-only files,
  so cherry-pick them onto each new `-lofyer` branch.

Note the backend `origin` points at the private registry and `upstream` at GitHub,
mirroring the frontend convention:

```
upstream  https://github.com/gpustack/gpustack
origin    ssh://git@192.168.0.23:11022/root/gpustack.git
```

## Brand rules

User-facing brand is **ZStack AIOS** (upstream **GPUStack**). Functional identifiers
are **not** rebranded, to preserve compatibility with upstream contracts across the
frontend/backend boundary:

- lowercase `gpustack` (npm/pip package, URLs, paths, k8s namespace)
- `GPUSTACK_*` env constants
- `X-*` HTTP headers, cookies (`gpustack_session`, `gpustack_oidc_*`)
- JS/Python identifiers (e.g. `getGPUStackPlugin`, `gpustack_worker` proctitles)
- backend-contract strings (see `SKIP_LINE_PATTERNS` in frontend `scripts/rebrand`)

The API key **prefix** (`zstack`) is a deliberate exception: it is user-facing
and, by our choice, not upstream-compatible.

## Shipping a customer build

A release requires **both** the rebranded frontend and the backend on matching
`v<version>-lofyer` branches. The deliverable is a single control-plane container
image built by `gpustack-lofyer/hack/package-lofyer`.

### Building the image

One command from the workspace root:

```bash
scripts/build-image        # frontend build -> image -> full verification
```

It resolves the version from the (matching) `v<version>-lofyer` branches, builds the
rebranded frontend, delegates to `gpustack-lofyer/hack/package-lofyer` for the image,
then runs `scripts/verify-image`. Flags: `--skip-ui-build`, `--no-verify`, `--push`,
`--dry-run`. Env: `NAMESPACE`, `REPOSITORY`, `TAG`, `IMAGE`, `BASE_IMAGE`,
`SMOKE_PORT`.

The backend script can also be used directly if you only want the image step:

```bash
cd gpustack-lofyer && ./hack/package-lofyer   # -> mesastack/gpustack:v<version>
```

It derives the upstream tag from the current branch name, so it always overlays a
matching official base image. Knobs: `UI_BUILD=1`, `DRY_RUN=1`, `PUSH=1`, `TAG`,
`IMAGE`, `BASE_IMAGE`, `NAMESPACE`, `REPOSITORY`, `UI_REPO`.

### Why a thin overlay, not `make package`

Upstream `pack/Dockerfile` rebuilds the entire control plane (PostgreSQL 17,
Higress gateway stack, Prometheus/Grafana, Skopeo compiled from source, ROCm
`amd_smi`, vLLM extras). Those components are hardcoded and cannot be skipped, so a
full `make package` costs 30-60+ minutes and tens of GB. Our fork only diverges in
two places, so `pack/Dockerfile.lofyer` overlays them onto the official release
image (`gpustack/gpustack:v<version>`) in seconds:

1. `gpustack/ui/` replaced with our rebranded frontend `dist/`
2. fork-patched Python sources (currently just `gpustack/security.py`)

The script stages only files that actually differ from the upstream tag
(`git diff <tag>..HEAD -- 'gpustack/**/*.py'`), keeping the layer small and
auditable. The Dockerfile self-checks the result at build time (asserts
`API_KEY_PREFIX == zstack`, and that the UI carries the ZStack AIOS brand).

Note this image is the **control plane** (server + worker). Inference runners use
separate `gpustack-runner` images pulled at runtime and are unaffected.

If you ever need a fully self-built image with no upstream base (clean supply
chain), use `PACKAGE_UI_DOWNLOAD=false make package` after staging our `dist/`
into `gpustack/ui/`, and expect the long build.

### Verifying a build

`scripts/build-image` runs this automatically; use it standalone to re-check an
existing image:

```bash
scripts/verify-image                 # static + runtime checks
scripts/verify-image --static-only   # skip booting a container
IMAGE=mesastack/gpustack:v2.2.3 scripts/verify-image
```

Checks performed:

- **static**: `gpustack version`, `API_KEY_PREFIX == zstack`, old `gpustack_`
  keys rejected, masked value format, UI assets present, UI carries the ZStack AIOS
  brand, no leftover `GPUStack` string in `index.html`
- **runtime**: boots with `--disable-worker`, confirms `/` serves
  `<title>ZStack AIOS</title>`, logs in via `POST /auth/login` (bootstrap password at
  `/var/lib/gpustack/initial_admin_password`) and creates a key via
  `POST /v2/api-keys` to prove the `zstack_` prefix is live end to end

The test container is always removed, including on failure.