---
icon: 🚀
---

# Cloud Deployment Paths

How code reaches `cloud.activepieces.com`. `continuous-delivery-cloud.yml` is the **Cloud gate** (ENG-1021): `cut` → `e2e-matrix` + `upgrade-test` → `report` + `tickets` → `approve` → `deploy` (migration check, canary, fleet) → `linear-release`. It runs by hand only; the Sunday-to-Thursday 08:00 UTC schedule returns once E2E coverage is broader.

## The normal path
A manual `cut` takes the tip of `main`, or the `sha` it names, and stops when that commit's `deploy-staging-app` and `deploy-staging-workers` check runs have not passed. A scheduled `cut` takes the newest `main` commit whose two check runs passed, and stops when the fleet already runs it. Every later job uses the `version.sha.beta` image that staging built from that commit; nothing is built again. The `approve` job holds the only approval (`production` environment). The `deploy` job then runs the breaking-migration check and deploys the canary and the fleet, a few minutes apart.

## Hotfixes
A hotfix merges into `main` first. A manual gate run with `hotfix_prs` (ENG-1098) cherry-picks only those PRs onto the commit Cloud runs, pushes `hotfix/cloud/<run id>-<attempt>`, builds `<version>.<sha>.pick` in `build-pick`, and sends it through the same checks, approval and `deploy`. Picks stack until the next normal run. No step checks that its candidate contains every picked PR, so the approver checks it. `emergency-cloud-deploy.yml` and `continuous-delivery-rollback.yml` stay outside the gate.

## Staging (upstream of the gate)
`continuous-delivery-stg.yml` builds every push to `main` and deploys it with **Kamal**, not Kubernetes: it SSHes to the devops box and runs `kamal deploy --config-file=config/{app,worker}.yml` from `/root/mrsk/stg`. App containers live on one host, workers on another; `kubectl`'s `stg` context on that box is dead and points at a node that no longer runs k3s — ignore it. Env vars for staging go in `config/app.yml` under `env.clear` (secrets are name-listed under `env.secret` and read from `.kamal/secrets`). The two deploy jobs' check runs on the commit are what the gate's `cut` reads. No E2E suite runs against staging.

## Gotchas
- **Kamal reads its config at deploy time, so editing `config/app.yml` during an in-flight CD run silently misses.** The build takes ~5 min and the deploy job reads the file when *it* starts; an edit that lands in between applies to neither the running containers nor the deploy. Worse, Kamal replaces containers one at a time, so a mid-deploy edit can leave app_1 and app_2 disagreeing about a flag — which reads downstream as a flaky feature, not a config race. Always re-run `kamal deploy --version <same tag> --config-file=config/app.yml --skip-push` after editing, and verify with `docker inspect <container> --format '{{range .Config.Env}}...'` on **every** container, not one. The command needs a TTY (`ssh -tt`); without it Kamal exits 0 having done nothing ("the input device is not a TTY").
- **The breaking-migration check runs in the `deploy` job, after the approval.** It fails when any pending migration carries `breaking = true` (rollback safety — see `tools/scripts/check-manifest-migrations.ts`), so the fleet never deploys. To ship a breaking migration on purpose, run the gate by hand with `skip_migration_check`.
- **`deploy` re-reads Cloud under the lock, but does not check that the candidate keeps what Cloud runs.** It fails when the canary and the fleet run different images, or when the fleet moved after `cut`. A check that refused a candidate missing a PR Cloud runs was removed in ENG-1483: it matched commits by `(#N)` subject, so it blocked a hand-made hotfix of a commit pushed to `main` without a PR, even though the candidate contained it. A scheduled `cut` takes the newest commit that staging deployed, which can be older than a hotfix Cloud runs, so the approver checks that the candidate contains every hotfix. After `emergency-cloud-deploy.yml` (fleet only), deploy the fleet's tag to the canary by hand or the gate stays blocked.
- **A manual run with an empty `sha` deploys the tip of `main` or nothing (ENG-1529).** Before 2026-11-18 it walked back to the newest commit staging deployed, like a scheduled run. That day the tip (a security fix) and its parent had failed the staging build, so the gate shipped an older commit and the captain thought the fix was on Cloud. Start the gate after the tip's staging deploy is green.
- **`cut` does not fall back to a green commit.** The E2E matrix runs on the one candidate only, so a red matrix stops the gate until `main` is fixed or a manual run names an older `sha`. Until 2026-11-18 `cut` read an `e2e-staging` check and skipped red commits; that job was removed because in its last week 16 of its 17 red runs were spec–staging mismatches, not product bugs, and each one stopped the Cloud pick.
- **A hotfix run has its own approval group (`cloud-approval-hotfix-<run id>`).** The full gate run's group cancels a waiting older run, which would otherwise cancel a hotfix waiting for approval. The re-read under the lock in `deploy` catches a hotfix and a full gate run that were cut from the same Cloud commit.
- **The hotfix push uses the release GitHub App token, not `GITHUB_TOKEN`.** A picked commit can change a workflow file, and `GITHUB_TOKEN` cannot push those. `RELEASE_APP_ID` and `RELEASE_APP_PRIVATE_KEY` must be repo secrets: `cut` has no environment, so the copies in the `release` environment are empty there.
- **Every job below `build-pick` needs an explicit status function in its `if`.** A normal run skips `build-pick`, and an `if` with no status function gets an implicit `success()`, which is false when any earlier job in the chain was skipped. The first rehearsal (2026-10-29) passed every check, then skipped `approve` and `deploy` and still showed green. A job that must not run after a failed check (`approve`, `deploy`, `linear-release`) uses `!cancelled()` plus `needs.<job>.result == 'success'`, not `always()`, so a cancelled run never deploys. A job that reports failures (`report`) keeps `always()`.
- **Tag readers accept `.beta` and `.pick`.** The gate and `continuous-delivery-rollback.yml` both parse the running tag; a reader that matches only `.beta` reads nothing after a hotfix deploy.
- **The approval sits before the canary on purpose.** The canary runs migrations on the shared database at boot, so a canary deploy that waits days for the fleet's approval brings back the ENG-1062 problem.
- **A reusable workflow checks out `github.sha`, not the candidate.** `e2e-matrix.yml` and `upgrade-test.yml` take a `ref` input, and the gate passes the candidate SHA; without it the specs come from the newest `main`.
- **The canary and the fleet deploy in one job on purpose.** GitHub keeps only one pending job per concurrency group, even with `cancel-in-progress: false`, so with a lock per job a third run could replace the second run's pending fleet deploy after its canary had migrated the shared database. The `deploy` job's group is `promote-production`, the same as `continuous-delivery-rollback.yml`, so a rollback cannot interleave with a gate deploy either.
- **`/flags` cannot prove the upgrade test swapped images.** After the release sync PR, `main` carries the same `package.json` version as the last release, so the upgrade test compares the running containers' image ID with the candidate's instead. Its `swapped` output tells `report` whether a red result is the candidate or the setup before the upgrade.
- **The canary has no daily schedule and no manual dispatch since ENG-1062.** It shares the prod database and Redis, so a daily build of `main` ran newer migrations under the older fleet (2026-10-06: days of old-shape writes). The canary app runs migrations at boot, so it now gets only the image of a Cloud deploy (since ENG-1021 it has no workflow of its own), and `continuous-delivery-rollback-canary.yml` (which ran `down()` on the shared database) is deleted. The Cloud rollback workflow does not touch the canary hosts, so after a rollback they keep the bad image until the next Cloud deploy.
- **`docker pull … error from registry: denied` on the ops host can be transient.** Seen 2026-08-12 pulling a `.beta` tag to the canary host; re-running the same job with no other change pulled and deployed fine. The tag existed in GHCR the whole time. Re-run once before suspecting the host's registry credential — GHCR does answer `GET /v2/` for a token that cannot pull, so login-succeeds/pull-denied is *consistent* with an expired credential, but it is not evidence of one.
- **Never put a BuildKit cache mount on `/var/cache/apt` or `/var/lib/apt`.** The `node:*-bullseye-slim` base ships `/etc/apt/apt.conf.d/docker-clean`, which wipes downloaded `.deb`s and sets `Keep-Downloaded-Packages "false"` — so the mount caches nothing, but it does persist stale `apt` lists and `partial/` leftovers across Depot builds. When `bullseye-security` republishes and old `.deb`s rotate out, the next build dies on `Hash Sum mismatch` / `Unable to fetch some archives` → `exit code: 100`, which reads like a missing package but isn't. Removed from both Dockerfiles on 2026-08-12; a plain `apt-get update && apt-get install` is what works. Also: bullseye `main` has been frozen since Aug 2025 and Debian 11 LTS ends Aug 2026, so the base image needs a bookworm bump.
- **Building the image locally from a branch can fail before your code is even copied.** Two independent causes seen 2026-09-10 on arm64: (1) `snapshot.debian.org` served corrupt `.deb`s (`Hash Sum mismatch`, `File has unexpected size … Mirror sync in progress?`, `Bad header line`) across three retries, with no cache mount involved — workaround is a throwaway Dockerfile whose base stage is `FROM ghcr.io/activepieces/activepieces:<root package.json version> AS base` (same bun/npm/isolated-vm tooling, no apt), plus `ENV NODE_ENV=development` and `RUN rm -rf /usr/src/app` at the top of the build stage (the published image sets `NODE_ENV=production`, which makes `bun install` skip devDependencies). (2) `bun install --frozen-lockfile` fails whenever any workspace `package.json` version differs from its `bun.lock` entry; check with `bun install --lockfile-only` in a copy and `diff` the result before blaming your change.
- **`breaking = true` on a migration and the `⛓️‍💥 breaking-change` PR label are different axes.** The migration flag is about rollback safety and is what stops deploys; the label is about self-hoster upgrade impact and is enforced by `breaking-change-check.yml` on PRs. Neither implies the other.
- **The Dockerfile's prune step deletes `packages/pieces/core` (and every community piece but the api's 5), then re-runs `bun install` without a lockfile.** Any other kept workspace that declares one of the removed pieces as `workspace:*` breaks the image build with `Workspace dependency "@activepieces/piece-…" not found`. #15842 hit this by adding core pieces to the engine's devDependencies for CI tests; the prune step now strips the engine's `@activepieces/piece-*` devDependencies. Check any new piece dependency outside `packages/pieces` against that step.
- **The run stage installs with `bun install --production`, so anything a kept workspace's compiled `dist` requires must be in its `dependencies`, not `devDependencies`.** The run stage copies `packages/` from the build stage, workspace `node_modules` links from the full install included, and bun 1.4.0 left those stale devDependency links in place, which hid that `tslib` was dev-only in `pieces/framework`, `pieces/common`, `core/ai-providers` and the api's 5 pieces, and that `server/api` listed those 5 pieces as devDependencies while importing them. bun 1.4.4 removes stale links (oven-sh/bun#45307), and the image died at boot with `Cannot find module 'tslib'` (seen 2026-10-27, also reported by a self-hosted customer building their own image). `bun install` in the repo stays green; only the built image shows it, and a new package that compiles with `importHelpers` reintroduces it (ai-providers did within days). After a bun bump, build the image and resolve every bare `require` in `packages/**/dist` from its own directory.
- **Caddy on the app hosts keeps about 15 minutes of access logs, so capture 502 evidence while the incident is live.** It logs to stdout through Docker's `json-file` driver with `max-size: 10m` and no `max-file`, so the single file is cut back each time it fills. On 2026-11-10 every app host's oldest Caddy line was ~12:17 UTC, two hours after a wedged-container alert, whose window was already gone. An app container's own log has the same 10 MB cap. Copy `docker logs` of the suspect container and Caddy to the DevOps box before recycling anything, because the next Kamal deploy prunes exited containers too.
