---
icon: 🧠
---

# AI Providers

Lets platform admins configure one or more LLM backends for AI pieces in flows. Also auto-provisions an "Activepieces" provider (backed by OpenRouter) whose credit balance and auto-top-up are metered by Autumn billing. Registered in every edition (`aiProviderModule` sits outside the edition switch in `app.ts`).

### Entities & services
- **AIProvider** — platform-scoped: `displayName`, `platformId` (indexed with `provider`, NOT unique — a platform can hold multiple keys per provider since the 2026-08 providers redesign), `provider` (AIProviderName enum), `auth` (EncryptedObject, AES-256 at rest), `config` (JSON), `enabledForChat`, plus per-key scoping: `modelScope` (`all|selected`) + `modelIds[]`, and `projectScope` (`all|selected|except`) + `projectIds[]` (GIN-indexed).
- Backend under `packages/server/api/src/app/ai/`; shared schemas in `core/shared/.../ai-providers/`.
- Supported providers (20): `openai`, `anthropic`, `google`, `azure`, `openrouter`, `bedrock`, `vertex`, `mistral`, `cohere` (native `@ai-sdk/cohere`), `cloudflare-gateway`, the OpenAI-compatible vendors `xai`, `deepseek`, `zai`, `qwen`, `minimax`, `moonshot`, `meta`, `nvidia`, `custom` (OpenAI-compatible, e.g. Ollama/LM Studio), and `activepieces` (auto-provisioned via OpenRouter).

### How it works
- `GET /` list (auto-creates ACTIVEPIECES when `aiCreditsEnabled`); `GET /:provider/models` (cached); `POST /` create (validates creds first); `POST /:id` update; `DELETE /:id`.
- For ACTIVEPIECES, update is blocked by an early return (only enabling `enabledForChat` is allowed); deletion is allowed and self-healing — `listProviders` recreates the managed row. The delete revokes the row's OpenRouter key, and the platform hard delete does the same through `deleteManagedProvider`, because the `ON DELETE CASCADE` from `platform` would otherwise leave the key live. The recreated row mints a fresh $500 key, so a limit raised by `increaseAiCredits` does not survive a delete.
- Engine integration: the AI piece no longer fetches credentials. It POSTs `/v1/ai/execute`, the server enqueues an `EXECUTE_AI` worker job, and the worker resolves the provider over the `resolveAiProvider` RPC. `GET /:provider/config` was removed in [#15494](https://github.com/activepieces/activepieces/pull/15494).
- For the managed ACTIVEPIECES provider that config route is also the **credit gate**: `assertCreditsAndAppSumoNotExceeded` (`platform/billing-provider.ts`) throws `QUOTA_EXCEEDED` when the Autumn credit or AppSumo balance is blocked — fires per AI call, but usage is only metered post-run, so in-flight spend is invisible to it. See decision 000016 (`brain/decisions/000016-managed-ai-metering-moves-to-centralized-worker-execution.md`).
- Activepieces provisioning: `getOrCreateActivePiecesProviderAuthConfig()` → `enrichWithKeysIfNeeded()` mints an OpenRouter key. No system job is scheduled — renewal/top-up is driven by Autumn (`autoTopUps` in `autumn-billing.ts`), not Stripe.

### AI Credits (Autumn-metered)
- Rate: 1000 credits = $1 USD; OpenRouter meters usage per API key; usage cached 180s.
- New managed keys are minted with a hard spend guardrail of **$500/month** (`MANAGED_OPENROUTER_KEY_MONTHLY_LIMIT_USD` in `ai-provider-service.ts`, `limit_reset: 'monthly'`) — a runaway-cost ceiling independent of the Autumn credit balance. Keys minted before the 2026-07 change carry $1000/monthly from a one-off OpenRouter backfill.
- There is no monthly credit-reset job and no direct Stripe invoicing; the only "monthly" mechanism is the key's `limit_reset`.
- Model lists are cached in memory, cleared daily at midnight via cron.

### Provider visibility
- `isActivepiecesAiProviderHidden` hides the managed provider when the `aiCreditsEnabled` flag is off (`OPENROUTER_PROVISION_KEY` unset — typical self-hosted) or when `shouldHideActivepiecesAiProvider` returns true: the plan includes Embedding and no self-serve trial is running (`billingProvider.getSelfServeTrialInfo` does not return `active`). A trial counts only until its end. The first check after the end starts one entitlements refresh, so the plan drops the trial's Embedding. The check reads the trial dates only from the Redis cache that the entitlements refresh fills, and an empty cache answers "no trial", which is main's behaviour. The Embedding page tells trialers that Embedding plans don't usually include Activepieces AI.
- Hidden means treated as absent everywhere: `listProviders()` omits the row and `getChatProvider()`/`getChatProviderName()` return null (`findAvailableChatProviderRow`). This keeps a stale `enabledForChat` (e.g. from the 0.82.1 migration) from pinning chat to a provider that 402s with no top-up path (GIT-1620).

### Model catalog

Per-model metadata (context window, max output, release date, per-million input/output price,
tool-calling / reasoning / vision) for the models a provider lists. Sourced from
[models.dev](https://models.dev) (MIT), generated by `npm run sync-model-catalog` and published by a
weekly workflow to `https://cdn.activepieces.com/ai/model-catalog.json` — **nothing is committed and no
process imports it**. `modelCatalog.lookup({ provider, modelId })` is the single accessor: it is async,
fetches the object once and caches it for 24h, dedupes concurrent callers on one in-flight promise, and
backs off for 5 minutes after a failure so a CDN outage cannot slow the models endpoint. Enrichment
happens once, in the `modelsCache` re-map inside `fetchModels`, so the web picker, the AI piece dropdown
and `ap_list_ai_models` all get it from the same place. See decision 000032.

Prices are rounded to three decimals in the generator, both to kill float artefacts
(`0.049999999999999996`) and because a handful of OpenRouter models — `deepseek/deepseek-v4-flash`
among them — carry continuously floating five-decimal prices. Three decimals is below anything the UI
renders and preserves every real price; the cheapest in the set is 0.01.

### Model tiers

The published tier file, `https://cdn.activepieces.com/ai/pricing.json`, carries two lists: **flow tiers**
(`tiers`, `defaultTierId`, `fastTierId`) for the surfaces that name their own model (FLOW_STEP and AGENT),
and **chat tiers** (`chatTiers`, `chatDefaultTierId`, `chatFastTierId`) for CHAT and AGENT_BUILDER. The
chat keys are optional; when absent, chat reads the flow list. Each list must contain its own default and
fast tier and have unique ids. There are no cross-list rules: the same id may sit on both lists pointing at
different models, and a tier may exist on one list only. A file that fails any rule is rejected whole and
the release's bundled tiers serve instead.

`modelTierCatalog.current(surface)` (`packages/server/utils/src/model-tier-catalog.ts`) is the single
accessor: synchronous, never throws, serves the cached file or `ACTIVEPIECES_CHAT_TIERS`, and refreshes in
the background every 15 minutes with a 5-minute back-off after a failure. `warmUp()` runs at API boot from
`aiProviderService.setup()`, bounded to 2 s so a dead CDN cannot block boot. The resolution family in
`ee/agent/agent-model-resolution.ts` (`findTier`, `resolveTier`, `resolveNamedModelId`, `resolveModelIdForProvider`,
`defaultModelIdForProvider`, `resolveModelIdForAnalytics`) and the fast-model helpers in `agent-helpers.ts` read it,
every one with a required `surface` argument. `surfaceOf({ source })` in that file is the only place the run
source maps to a list (FLOW_STEP and AGENT to flow, CHAT and AGENT_BUILDER to chat), and `agent-config-rpc.ts`
derives `namesItsOwnModel` from it. The AI piece's text steps resolve through `aiModelResolution.resolveTierModelId`
(`src/app/ai/ai-model-resolution.ts`, CE), always on the flow surface, in `ai-execute-controller.ts` before the job is
enqueued. `AP_MODEL_TIERS_URL` overrides the URL. The file is published from the console in `ap-analytics`; see
decision 000041. The console publishes both lists and both fast pointers on every Publish, and on its first load
after a file without chat keys it seeds the chat list from the live flow list.
### Platform tiers

**Platform tier** — an admin-built, platform-owned list of up to 5 `{configId, modelId}` entries (main + ranked fallbacks, any keys), table `platform_model_tier`, API `/v1/platform-model-tiers`. A tier follows key scope: in a project it runs only the entries whose key serves that project, and it is listed there only when its main entry can run; see decision [000048](../decisions/000048-platform-tiers-follow-key-project-scope.md) (replaces 000044's grant). The admin Tiers page (`/platform/ai/tiers`) is listed in the AI Center sidebar and global search on EE and Cloud, and hidden on CE like Usage. _Avoid_: "model tier" alone — that is the console **credits tier** (`ModelTier`, above).

**Specific model** — a model a builder picks directly from one key (`{provider, configId, model}`), as opposed to a tier. The admin switch `platform_configuration.aiSpecificModelsVisible` (Tiers tab, "Visible to builders") hides them from builders. _Avoid_: "Advanced" (an early mockup word for the same section)

### Gotchas
- **LiteLLM in front of a non-OpenAI model breaks `@ai-sdk/openai` Responses on non-streamed calls.** LiteLLM's Anthropic bridge returns `reasoning` output items without `summary`, and the SDK's schema requires it: `AI_APICallError: Invalid JSON response`. It shows up on a second turn, when Claude thinks; single-turn and streamed calls pass. Chat Completions never returns reasoning items, so CUSTOM on its default Chat style works through LiteLLM while CUSTOM on Responses hits this; so LITELLM is Chat Completions only. A `400 No connected db.` from LiteLLM means the key sent is not its master key and the proxy has no database for virtual keys; check the saved key before anything else. Seen on LiteLLM v1.106.6 and v1.108.0. Images through LiteLLM: `createOpenAI().image()` sends `response_format: b64_json` for any id not starting with `gpt-image` (aliases, `openai/gpt-image-…`), and LiteLLM rejects it unless `drop_params` is on. `createOpenAICompatible().imageModel()` generates and edits fine. A LiteLLM wildcard route (`openai/*`) expands to 200+ rows in `/model/info` and `/v1/models`, many not callable (`openai/low/1024-x-1024/gpt-image-1`), and no field marks them as expanded. Test bench: ENG-1510 `compat.md`.
- **A LITELLM key fills its model list from the proxy on save and on refresh (`litellmDiscovery`).** It reads `/v1/model/info` at the proxy root (a trailing `/v1` on the base URL is stripped) and only falls back to `/v1/models` on a non-auth HTTP error, because only `/model/info` carries `mode`, prices and capability flags. Rows marked `discovered: true` are replaced on every sync; rows without it are hand-typed and always kept. A blocked or unreachable proxy still lets a save through when models were typed in, but a 401/403 never does, and refresh always fails loudly. A missing `supports_function_calling` stays unset, not `false`, so the model stays visible in the tool-model picker.
- **The connect dialog marks the failing field by matching the server error text.** `providerCredentials.fieldOfServerError` (web) matches the messages from `litellmDiscovery` and `assertUsableHeader` with regexes: "rejected the API key" goes to the API key field, "blocked the request to" and "could not list the models" go to Base URL, and "API key header cannot be sent" goes to the key header. If you reword one of those server messages, update the regex too, or the field stops turning red and only the banner shows. The unit test `provider-credentials.test.ts` keeps copies of the messages.
- **A LiteLLM proxy on your own machine is blocked by the SSRF filter.** Connecting `http://127.0.0.1:4000/v1` fails with "points to a private or internal address" until the server starts with `AP_SSRF_ALLOW_LIST=127.0.0.1`. Put it in `.env.dev`, which the API loads itself. A shell variable alone is dropped: turbo runs in strict env mode and the variable is not in `turbo.json`'s `globalPassThroughEnv`, so it needs `AP_SSRF_ALLOW_LIST=127.0.0.1 npm run dev -- --env-mode=loose`. Without the `--`, `--env-mode=loose` goes to npm, not turbo.
- **The key page hides Remove on synced LiteLLM rows.** A `discovered` row deleted in the UI comes back on the next save, so `ManualModelList` disables Remove and the type toggle for it; the fix is in LiteLLM. Hand-typed ids enter through the connect form's `typedModels` field, which `providerRequest` turns into TEXT models and never sends as config.
- **On a key with listed models, a chat tier finds its model by `metadata.realModelId`, not by id.** LiteLLM aliases (`ap-fast`) hide the real model, so `resolveModelIdForProvider` picks the listed text model whose `realModelId` contains the tier's native model (`claude-haiku-4-5` for `fast`). No match falls back to the first listed model. Spend headers (`x-litellm-trace-id`, `x-litellm-tags`, and `x-litellm-customer-id` on flow steps only) go to `LITELLM` only, via `aiUtils.litellmSpendHeaders`.
- **Image models skip the text path's wrappers.** `billedLanguageModel` never sees an image call, so a provider cost header only reaches AI Usage because `getGeneratedImage` reads `generateImage().responses[].headers`. Key health and request headers only reach image calls because `aiUtils.createModelForImages` passes them into `createImageModel` as `options`. A new provider or header has to be added to the image path separately.
- **The Activepieces credits row is created lazily, so any read of provider rows must call `ensureManagedProviderRow` first.** `listVisibleRows` and `findAvailableChatProviderRow` both do. A brand-new platform has no row until one of them runs. Before AI-173, agent creation read the table directly, so a new cloud signup whose first AI action was "New agent" got an agent with no model, and its first message failed with a 409. A test that seeds the row with `mockAndSaveAIProvider` can't catch this, so cover the fresh-platform path without seeding.
- **Every AI Center page links its admin guide from the header, through `adminPageResources` (`packages/web/src/app/components/admin/admin-page-resources.ts`).** `aiProviders` → `admin-guide/guides/setup-ai-providers` (Providers and the key page), `aiTiers` → `ai-tiers`, `aiCapabilities` → `ai-capabilities`, `aiUsage` → `ai-usage` (overview, list and detail). A new AI Center page needs a guide under `docs/admin-guide/guides/`, an entry in the docs.json **AI Center** group, and a `resources=` on its `AdminPageHeader`.
- **A new provider shows "—" for context and price until the catalog is republished from `main`.** Its `MODELS_DEV_PROVIDER` entry only takes effect when the weekly `sync-model-catalog` workflow runs on `main` (trigger it by hand after the merge), and the API caches the catalog for 24h. Even then only ids models.dev knows fill in: of NVIDIA's 85 hosted models about 26 match, and its prices are $0 because the hosted catalogue is free.
- **The AI usage read API (`/v1/ai-usage/*`) works in UTC days and scales today's partial day inside the SQL.** Every previous-range sum is multiplied by a weight that is below 1 only on the day matching today, so totals, cards, lists and related items all compare the same way. Scaling after the query, in TypeScript, can't reach per-item numbers and makes the views disagree.
- **Usage tracking starts per platform, at its copy cursor.** `trackingSince` is the cursor's `startedAt`, or the platform's creation if that is later. A period that starts earlier hides its change %, so a fresh platform never shows fake growth. A platform without a cursor has no tracking start.
- **Usage ids are not always entity ids.** `no-project` is platform chat, `activepieces-credits` and `unknown-<provider>` are keys with no config, and a model id is `base64url(provider|modelId)`. Lists stop at 1000 rows. `ACTION_RUN` rows have no `flowId`, so the flows card never adds up to the flow-steps slice.
- **AI usage analytics come from Autumn events, not our own stream** (decision 000053). Every Autumn `track` for an AI call carries a `usage` block v1 (`ai/usage/ai-usage-block.ts`). The block is built inside `tryCatch`: if pricing fails, the billed track still goes out without it, so billing never depends on analytics.
- **Usage-only calls send a value-0 `apCredits` track with only `{ usage }`.** That covers BYOK chat and agent turns, compaction, memory, personalization, own-key embeddings and managed calls without a cost. Source is `ai_usage`, with no top-level `projectId`, so the billing page's `groupBy projectId` isn't crowded. The value is a literal 0: `chargeFor` would return `toolCalls`.
- **`AI_USAGE_COPY` reads a delayed, non-overlapping window.** Each run reads `[copiedUntil, min(now − 2 min, copiedUntil + 10 min))`. The 2 minutes cover Autumn's 10–20 s visibility lag. The cursor moves with a compare-and-swap on epoch milliseconds in the same transaction as the upserts. The run's end time is fixed once, at its start: chasing `now` loops on tiny windows until the budget runs out. A window that needs more than 3 pages is halved and retried (down to 1 s, which reads uncapped), and the run's 40 s budget is only checked between windows: aborting mid-window would re-read the same busy window forever. The smaller window is saved in Redis per platform for an hour, so a run that spends its budget on a capped read still leaves the next run a smaller window instead of the full 10 minutes again. A full-size window that reads in one page doubles back toward 10 minutes (the short leftover window at the end of a run never grows it, or a busy platform would grow and re-halve every run), and the Redis entry is deleted at full size, so one busy spell does not leave a platform on tiny windows. A bad event must be skipped at parse time (token counts are integers, the block is run through `sanitizeObjectForPostgresql`): one row Postgres rejects fails the whole window's write and pins that platform's cursor until someone fixes it.
- **Autumn drops `null` values from event properties, including inside the nested `usage` block.** The stored event has no `agentId` or `userId` keys at all. The block's id fields therefore accept missing and turn it into `null` (`nullish().transform`). A schema that requires the key rejects every event, and the cursor still moves past them, so the page stays empty with no errors.
- **`events.list` isn't fail-open and can only filter by feature.** It throws on 5xx or timeout, so the job wraps it and retries the window next minute. It also returns flow-run, MCP and chat-turn events; the job skips anything without a `usage` block.
- **One Autumn customer can hold many platforms, so the copy job keeps only events whose `usage.platformId` matches.** Every platform that activates the same license key shares one Autumn customer (on Cloud in Nov 2026, 11 customers were shared by 2 to 32 platforms). `events.list` returns the whole customer, and usage-only events carry no top-level `platformId`, so the platform id must live inside the block. Before this filter, each platform showed every sibling's usage as "Deleted project/user" and the rollups were wiped (ENG-1417).
- **The usage page's pace divides by tracked time, which starts at the cursor's `startedAt`.** On day-length periods "Average per day" divides by at least one day, so a platform tracked for a few hours does not show a projection above its total; 1h–24h periods show a "Daily run rate" projection on purpose. Today's bucket is unfinished, so the chart draws the last segment dashed while the period is running (ENG-1509).
- **Runs are counted through `ai_usage_run` (run ids kept 24 h).** The earliest event of a new run gets `runs = 1`. A run whose AI calls are more than 24 h apart counts twice.
- **models.dev already has cache prices (`cost.cache_read`, `cost.cache_write` for a 5-minute write) and long-context `tiers` for most Anthropic, OpenAI and Bedrock models.** `tools/scripts/sync-model-catalog.ts` has to copy them, or every estimate ignores caching. It has no batch, flex/fast or 1-hour cache-write prices; LiteLLM's `model_prices_and_context_window.json` (MIT) does. We send no `serviceTier`, so every call bills at the standard rate.
- **`billing` is a required argument of `createModel`, `createEmbeddingModel` and `searchWeb`.** A call that is really unbilled passes `null`, so a new call site can't silently drop out of usage.
- **The SDK's input token total already includes cache reads and writes.** The estimate subtracts both before pricing uncached input.
- **Production runs are queued, so their `flow_run` row may not exist yet.** `aiCallContext.runOf` reads the queued-run metadata hash in Redis first, then the saved row, and caches the result in-process. MCP, trigger and property executions use placeholder run ids (`DEFAULT_MCP_DATA.flowRunId`, `execute-trigger`, `execute-property`). They become `ACTION_RUN` and never count as runs.
- **`enabledForChat` now means "the Default AI key", not "the key every chat runs on".** Chats pin their own key (see `ai-agents.md`); the flag only seeds the legacy and default paths, knowledge-base embeddings (when no Embeddings capability is set), and platform-wide work. Memory and personalization run on it, else on the Default tier's main model when that key serves every project, else on Credits (`agentHelpers.resolvePlatformWorkModel`). `listForProject` still reports the flag so a web client from before the 7b deploy keeps working; no current web code reads it.
- **Embedding models are a short curated list per provider (`EMBEDDING_MODELS` in core-piece-types), first entry = default.** The KB column is `vector(768)`, and embeddings are truncated to 768 dims, which is only safe for Matryoshka-trained models, so a live provider listing is not used. Embedding ids are never in a key's chat model scope, so the Embeddings capability skips `keyOffersModel` / `losesModel`. **Changing the Embeddings key or model leaves old files on their old vectors:** search mixes two embedding spaces until those files are re-uploaded. There is no per-file pin and no re-embed.
- **The replay eval can't judge a model swap where the new model takes a different tool path.** Two separate things cause it:
  - **Replay divergence.** Every fixture gets the same tool set from `buildEvalToolSet` (local, display, cross-project and thinking tools). The recording only fixes which results come back and in what order. Calling a registered tool out of the recorded order returns `__evalDivergence`.
  - **Tools the eval doesn't have.** The flow and table tools (`ap_duplicate_flow`, `ap_build_flow`, `ap_find_records`) aren't in that set at all, so `ap_get_tool_schema` answers "There is no tool named". The tool does exist in production.

  Either one makes a candidate give up for reasons that aren't the model's, so `--candidate-model` scores read low. On 13 Oct, Sonnet 5.5 scored 9/26 against 14/26, and 4 of its 6 regressions were one of these two. Use `--baseline-model`/`--candidate-model` for cost and for judge-only cases. Check tool-heavy behaviour with `agent-evals:live` before changing a tier.
- **Change which model a tier runs in the AI Tiers console, not in code.** A model a tier points at runs on credits without being in `ANTHROPIC_OPENROUTER_CHAT_MODELS` (`assertManagedModelAllowed` allows every tier `modelId`). Keep Claude Sonnet 5.5 and Opus 5.5 out of that list: everything in it counts as reasoning-optional (`canDisableReasoning`), and their OpenRouter endpoints reject `reasoning: {enabled: false}` with "Reasoning is mandatory for this endpoint", which would fail the thinking-off first step of every turn. Unlisted, they get `effort: minimal`, which works. Haiku 5.5 accepts `enabled: false` (checked 13 Oct).
- **New shared AI-provider schemas go inside `packages/core/shared/src/lib/management/ai-providers/index.ts`, not in a sibling file that imports from it.** `index.ts` re-exports its siblings, so a sibling that imports `AIProviderModel` back from `./index` gets `undefined` when the module loads, and every importer crashes with `Cannot read properties of undefined (reading 'optional')`. `platform-model-tier.ts` works only because it imports from `core-utils`.
- **The admin Tiers tab applies a key's `modelScope` itself.** `GET /v1/ai-providers/configs/:id/models` returns the unfiltered list on purpose (admins pick the allow-list from it), so the tier pickers and the Specific models list narrow it client-side in `modelMeta.scopedTextModels` (`packages/web/src/features/agents/ai-model/`). The server still refuses an out-of-scope entry on save (`assertEntriesValid`), so the client filter is convenience, not the guard.
- **A key a live platform tier uses cannot be deleted, and its `modelScope` cannot drop a tier's model.** `aiProviderService.delete` and `update` call `platformModelTierService.assertKeyCanBeDeleted` / `assertKeyScopeKeepsTiers` under a `platform` row `FOR NO KEY UPDATE` lock and answer 409 with the tier names. The same lock serialises every tier write, so a tier save can't race a key delete into a dangling entry. The same update also refuses removing a typed-in model (Vertex, OpenAI-compatible, Cloudflare `config.models`) that a tier uses, even when `modelScope` is `all`, and limiting the projects of a key that a capability (web search, scraping, image) uses, or dropping the capability's `modelId` from its model list: capability keys must serve every project and keep their model (`assertCapabilitiesKeepKey`).
- **A key's model list is enforced when a specific model runs, not only in pickers.** `/v1/ai/execute` checks `aiProviderService.assertModelAllowed` before queueing (the managed Activepieces key is skipped: it allows every model), and the worker's `resolveAiProvider` RPC sends `modelId` so the server checks it again against the key it resolves at run time (a model removed while the job waited fails then). `resolveNamedModelId` refuses any own-key model outside the list with no tier-id exemption: an own key never translates `smart`/`fast`, so a name that looks like a tier is just a customer model id. At platform scope, `rankRows` prefers keys open to every project; at project scope it still prefers the most specific key. Key edits and capability choices both take the `platform` row lock, and a capability save re-reads its key under that lock, so the two cannot interleave. One helper, `aiKeyScope.keyOffersModel`, decides whether a key still offers a model: its model scope, plus its typed-in `config.models` list when it has one. The tier and capability guards on a key edit both use it, and so does the locked re-check when a capability is saved. A key edit that removes a typed-in model a capability uses is refused like a model-scope change.
- **Deleting a platform tier needs a replacement, except the last live tier, and there is no usage count.** Refs to a deleted tier are never rewritten; they reach a live tier through `replacedBy` (flattened to one hop on delete). The last tier may always go with no replacement: without that exit, a platform's last tier and the key it uses could never be removed, because the key guard refuses keys a live tier uses. Tier delete is also deliberately **not** plan-gated (writes are), so a platform that downgrades can still clear its tiers and then remove its key. Its refs then point at a tier with `replacedBy: null`, and the resolver must fail them with a clear "tier removed" error. An exact "used by N flows" count needs a `LIKE` scan over every flow version's `trigger` text, plus agent jsonb and conversations, on the whole platform. We dropped it rather than pay that. If counts come back, add a denormalized `flow_version.modelTierIds` array + GIN index (the `connectionIds` pattern) instead of scanning. Deleting the last live tier also sets `platform_configuration.aiSpecificModelsVisible = true` in the same transaction, so "hidden needs at least one tier" holds without the client flipping it first (a client-side flip left models visible when the delete then failed, #16116). The other direction holds too: `platformConfigurationService.update` saves every field in one transaction that first calls `platformModelTierService.assertSpecificModelsVisibilityAllowed` (same `platform` row lock, refuses a hide when no live tier is left), so a refused hide changes no other setting either, so a concurrent last-tier delete and hide cannot leave builders with nothing. `PlatformConfigurationSettings` (the infra Configurations form) does **not** carry the flag; only `UpdatePlatformConfigurationRequestBody` does, so saving that form never rewrites it.
- **The Tiers tab re-implements the server's project-scope rule on the client.** `modelMeta.keyServesProject` / `entryRunsIn` mirror `rowAllowsScope` + `scopeAllows`, and power the per-tier "Not available in N projects" lines, the *Preview as project* select and the confirm dialog when a key's projects are narrowed (`modelMeta.keyScopeImpact`). Change one side and change the other, or the admin sees a tier as available where the server refuses it.
- **The keys page mirrors the tier-model rule above in the UI.** `keyScopeImpact` only checks project scope, so the key's model lists (`KeyModelList`, and `ManualModelList` for typed-in models) lock tier-used rows ("Used by {tier}") and always keep their ids in the draft, in both modes. Stats come from `AIProviderModel.metadata`; models outside the catalog show `—`.
- **On the key page the "Add new {provider} models automatically" switch *is* `modelScope: 'all'`, and every model checkbox stays live.** Unticking any model in `all` mode moves the draft to `selected` with every other listed model (plus tier-locked ones) in `modelIds` and turns the switch off. Nothing else appears or disappears on that toggle: an extra note line there shifted the table and was removed, so keep the switch state as the only signal. Ticking every model by hand, or the table-header checkbox, never turns `all` back on: only the switch does. Nothing stores exclusions; `all` means "everything, including future models", `selected` means exactly `modelIds`. The transitions live in `keyModelSelection` (`key-model-selection.ts`). The old *All models* / *Only selected* dropdown rendered disabled ticked boxes under `all`, and admins could not tell why they could not untick one. The header checkbox acts only on the visible rows (search, Capabilities and Selected only narrow them, and models behind *Show more* are left alone), the same as the Access tab. Acting on every filtered model let one click clear models the admin never saw.
- **The live-name unique index is `lower("name")`, which `EntitySchema` cannot express.** The entity declares it `synchronize: false` (so `check-migrations` skips it), which means test DBs built by `synchronize` don't have it. The case-insensitive check that tests rely on is the service's `assertNameIsFree`, which is race-free because it runs under the `platform` row lock; the index is the backstop.
- **`lockPlatform` takes `FOR NO KEY UPDATE`, not `FOR UPDATE`.** Every FK insert that references `platform` (projects, users, …) takes `FOR KEY SHARE` on the platform row, which `FOR UPDATE` blocks until commit. `FOR NO KEY UPDATE` serialises tier writes just the same without stalling those inserts.
- **Tier entries go through the same project check as specific picks.** `platformModelTierService.getForRun` takes a `ProviderScope` and applies `aiKeyScope.rowAllowsScope` + `scopeAllows` to every entry; if the stored main entry fails, the whole tier is unavailable (it is checked before the healthy-first sort, so a down main doesn't change the answer). The `configId`s still come from a live tier on the caller's platform, never from the caller. Only the label lookups (`getManyForRun`, `firstCandidates`) use platform scope, because they return no credentials.
- **Agent and chat runs read the tier grant from the conversation row, never from the worker.** CHAT and AGENT_BUILDER take `modelTierId` from the row (set by create/update). AGENT and FLOW_STEP take it from the job and write it onto the row **after** `acquireStreamingLock`, including null. Tool RPCs then call `aiModelCandidates.grantedEntryConfig` with the run's project, which honours a worker-sent `providerConfigId` only as an exact `{configId, modelId}` entry of that tier or of the Fast tier that runs in that project. A tier row switches projects through `agentModelTier.assertProjectSwitchKeepsTier`: every key the run can use (after the tool-calling filter, Fast tier included) must serve the new project. Clearing the project counts as moving to *every* project, so it needs keys open to all projects; a switch from no project checks the tier's platform-wide entries, because nothing records which keys the run kept. Any other `providerConfigId` goes through the normal scope check.
- **A platform tier step falls back only on provider failures, decided by `isFallbackWorthy` on a signal from `aiProviderSignal.fromError`.** 401/402/403/404/408/429/5xx, network errors (ai 7 throws them as `APICallError` with **no status** and `isRetryable: true`) and billing bodies (Anthropic sends low credit as a **400**) fall back; any other 4xx and our own errors (classify mismatch, no structured data, web search unsupported) stop. No fallback once a web-search step finished, since it was already billed.
- **Never flatten a provider error with `String(error)`; use `toError` from `@activepieces/core-utils`.** A provider can end a stream with a plain object instead of an `Error`. `String()` turns it into `"[object Object]"` and drops the status, so `isFallbackWorthy` has nothing to judge, the tier never moves to its next model, and the user sees `[object Object]` (incident 20 Oct 2026, AI-63). `toError` keeps the raw value as `cause`, which is what lets `aiProviderSignal.fromError` reach the status.
- **Worker-built models report no key health through `createModel`: no reporter is installed in the worker.** Tier steps report explicitly via the `reportAiKeyOutcome` RPC, and only provider signals (`fromProvider`) plus recoveries of non-active keys. Reporting our own errors would demote healthy keys to `unreachable`. The API never writes a worker-reported status for most providers: `confirmReportedOutcome` runs in the background and only triggers a `recheck` of the real key (once per key per 10s) when the report disagrees with the stored status. The exception is providers flagged `validationSkipsModelEndpoint` (CUSTOM, Vertex): their check never reaches the model, so a recheck would always pass, and the report is recorded as-is. Any direct `aiProviderHealth.record` from concurrent callers must pass `expectVersion` (the `statusVersion` it read), or an older report landing late overwrites a newer one. The worker token is shared and a shared worker serves every platform, so a report can't be bound to a platform; a forged one can at most cause a recheck. Don't pass `platformId`/`providerConfigId` to `createModel` in the worker, or a future reporter double-counts.
- **`ExecuteAiJobData` is a zod schema that strips unknown keys** (`JobData.parse` in `worker.ts` and `job-broker.ts`). A field added only to the TS type silently never reaches the worker; add it to `AiStepJobBase`.
- **Builders read the slim tier list (`GET /v1/platform-model-tiers`) with no `configId` or key names; the full entries are `GET /admin` only.** Don't widen the builder view: it would show project users the ids of keys scoped away from them.
- **The managed key only runs models the tiers or the chat allow-list permit, enforced in `aiUtils.createModel`.** The managed provider is OpenRouter underneath, so before this guard a crafted step naming any OpenRouter model ran on our key; the picker hid it but nothing denied it. `assertManagedModelAllowed` (`server/utils/src/ai-utils.ts`) checks `managedChatModelIds()` plus both surfaces' published tier `modelId`s and throws a `VALIDATION` error. Image models are checked too, in `createModel` (`imageGeneration: true`) and `createModelForImages`: only `ACTIVEPIECES_IMAGE_TIERS` model ids, plus the id the chat path passes as `adminChosenImageModelId` (the server-chosen Capabilities model). The worker now calls `modelTierCatalog.warmUp()` at start so a cold worker does not judge a newly published tier against the bundled list.
- **A catalog change takes up to ~2 days to reach a dropdown, through three caches in series.** CDN edge (`--cache-control max-age=3600`, 1h) → the server's in-memory catalog (`CATALOG_TTL_MS`, 24h) → `modelsCache` (flushed by the nightly `cron.schedule('0 0 * * *')`). So "I republished but the UI still shows the old price" is expected, not a bug; restarting the API short-circuits the two in-process layers. The edge TTL is deliberately 1h and not the 604800 that `publish-embed-sdk.yml` uses — that path is version-stamped, ours is a stable key rewritten in place, so a week-long edge cache would pin stale prices for a week.
- **`Sync AI Model Catalog` is the only workflow that proves the CDN S3 credentials still work — a green `Publish Embed SDK` proves nothing.** Both read the same `CDN_S3_ACCESS_KEY_ID` / `CDN_S3_SECRET_ACCESS_KEY` / `CDN_S3_BUCKET` / `CDN_S3_ENDPOINT` secrets, but `publish-embed-sdk.yml` guards its upload job on `needs.check.outputs.publish == 'true'`, which is false whenever the version in `packages/ee/embed-sdk/package.json` is already on the CDN. Its daily cron then finishes in about nine seconds with the upload skipped and the run still marked success, so the key can rot for weeks behind a green tick — that is how the model-catalog cron came to fail with `InvalidAccessKeyId` on three consecutive Mondays while the embed SDK looked healthy. To date a dead key, find the last embed-SDK run whose `publish` job actually ran (`gh run list --workflow=publish-embed-sdk.yml`, look for a duration in minutes rather than seconds) and compare it against the first model-catalog failure. To test a rotated key, dispatch `sync-model-catalog.yml` — it uploads unconditionally.
- **Publishing is an overwrite of one S3 key, and it only ever happens on the Monday cron or a manual `workflow_dispatch`.** Never on merge, deploy or release. There is no versioning and no history: last write wins and the previous contents are gone, which is why the object carries `generatedAt` — `curl -s https://cdn.activepieces.com/ai/model-catalog.json | jq .generatedAt` is the only way to tell how fresh what you are serving is. Editing the generator changes nothing in production until someone dispatches the workflow.
- **No egress to the CDN means no model metadata, permanently and silently.** Air-gapped installs, networks with an outbound allowlist, and CDN outages all fall back to the plain `{ id, name }` row with no message explaining it — every metadata field is optional, so nothing throws. `AP_MODEL_CATALOG_URL` points at a self-hosted mirror and is the only fix. This is a knowing exception to `.claude/rules/self-hosting.md` (decision 000032), so treat "self-hoster says prices are missing" as a network question, not a bug.
- **A new provider needs an entry in the generator's `MODELS_DEV_PROVIDER` map or it silently ships with no metadata.** models.dev provider ids do not match ours: `bedrock` → `amazon-bedrock`, `qwen` → `alibaba`, `moonshot` → `moonshotai`, `activepieces` → aliased onto `openrouter` at lookup time. The six OpenAI-compatible vendors were merged before the map was updated and produced exactly this — the run prints `no upstream source: …`, which is the thing to read after adding a provider. `cloudflare-gateway` and `custom` legitimately have no source.
- **The catalog object must be published before the code that reads it ships.** There is no bundled copy, so until a `workflow_dispatch` run puts it on the CDN, every install — including local dev — shows no metadata at all.
- **The pricing console shows bundled constants when no file is published, and says nothing.** `overview()` in `ai-pricing-service.ts` has three states, and two of them render the same numbers: **absent** (`NoSuchKey`) silently seeds the editor from `DEFAULT_PRICING_DRAFT`, the console's own checked-in copy of the tiers and weights, while **unreadable** (any other S3 error) shows those same constants *plus* a banner and blocks publishing. Only **present** is live data. So a screen full of plausible tiers is not evidence that anything was ever published — `curl -I https://cdn.activepieces.com/ai/pricing.json` is. The absent state is deliberate, it is how the first publish bootstraps, but it means "I can see the tiers" and "the prices are live" are unrelated facts. Worse, **absent and misconfigured are the same state**: `cdnClient.getJson` catches `NoSuchKey` and returns `undefined`, so a wrong bucket or an endpoint with the bucket in its path reads as "nothing published yet" and hides behind the defaults. The tell is the History tab — `list` does *not* catch `NoSuchKey`, so the same misconfiguration that reads as empty on the pricing page returns a 500 there. A 500 on history with a normal-looking pricing page means the S3 addressing is wrong, not that the history is empty.
- **Deleting the live tier file is safe for customers and confusing for everyone else.** `ai/pricing.json` was deleted on 2026-09-24 to drop a leftover test tier, and Activepieces kept serving the right tiers throughout: the 404 makes `fetchTiers` throw, the failure is logged and backed off, and `current()` keeps serving the last good copy the process fetched; only a process that never fetched one, a fresh boot during the outage, runs the `ACTIVEPIECES_CHAT_TIERS` shipped with the release. Two things do change. Each server logs 'Failed to refresh the AI model tiers file' (or 'Failed to load…' on a process that never got a copy) on every failed attempt, and attempts happen at boot and whenever `current()` is called with a stale or missing copy, at most once per five minutes (`FAILURE_BACKOFF_MS`), so a permanent 404 is retried the same way as a blip. And the console reads the missing key as **absent**, so the next publish starts again at `version: 1` while the `ai/pricing/history/` objects from the old numbering are still there — the version number is `live.version + 1`, not a counter, so it is only meaningful between two publishes that never lost the live key. To remove one bad tier, edit and publish; do not delete the object.
- **`CDN_S3_*` and `AP_S3_*` are two different buckets on two different providers, and mixing them puts public objects in customer data.** `cdn.activepieces.com` is a **DigitalOcean Spaces** bucket, `activepieces-cdn` in `fra1`, fronted by Cloudflare — a 200 from it carries `x-rgw-object-type` and `x-do-cdn-uuid`, which is how you tell. That is where `ai/model-catalog.json`, `ai/pricing.json` and `sdk/embed/*.js` live, and the credentials are the `CDN_S3_*` GitHub secrets. `AP_S3_BUCKET` is something else entirely: the **Cloudflare R2** file-storage bucket (`ap-files-prod` in production) holding run logs, step checkpoints, webhook payloads and flow bundles. The names are similar enough that the pricing console was once configured with the R2 file-storage key by mistake, which would have written a `public-read` `ai/pricing.json` into the customer-data bucket while `cdn.activepieces.com` stayed 404. When wiring anything that publishes to the CDN, confirm the endpoint is the Spaces region origin (`https://fra1.digitaloceanspaces.com`, never with the bucket in the path — the SDK appends `Bucket` itself and the doubled path returns `NoSuchKey` on every call).
- **The weekly sync workflow now pages Discord on failure, but the alert needs `DISCORD_ON_CALL_WEBHOOK` in this repo.** `sync-model-catalog.yml` ends with the same `if: failure()` step as `release-pieces.yml`. Before that it had no alert, and the only signal that it broke is a 404 on the object, which every reader swallows. Its run on 2026-08-31 failed with `InvalidAccessKeyId` — the `CDN_S3_ACCESS_KEY_ID` secret in this repo does not match a live key — so `ai/model-catalog.json` has never actually landed. Check `gh run list --workflow=sync-model-catalog.yml` before believing the CDN is the problem. **That same `CDN_S3_*` credential also publishes the AI tier file `ai/pricing.json` from `ap-analytics`** (decision [000039](../decisions/000041-ai-model-tiers-are-published-to-the-cdn-from-a-second-console.md)), so one dead key silently breaks model metadata *and* the published tiers at once: the console hides model costs (`model-catalog-service.ts` logs 'model prices will be hidden' and returns an empty list) while Activepieces keeps serving the tiers bundled in the release. Two features, one credential, no alarm on either.
- **A fresh Cloud platform cannot connect any AI provider, and the UI says nothing about why.** The admin page at `/platform/ai` sets `allowWrite = platform.plan.aiProvidersEnabled`, and that column defaults to **false** on the `free` plan (migration `1776…AddDefaultToAiProvidersEnabled`), so the connect button is simply absent rather than disabled-with-a-reason. On Cloud the flag is owned by Autumn (`autumn-utils.ts` lists it among the synced features), so there is nothing to configure locally and no `DEV_ENTERPRISE_PLAN` escape hatch on main. For local Cloud testing, flip it directly: `UPDATE platform_plan SET "aiProvidersEnabled" = true`. On EE or Cloud with no license, the throttled Autumn entitlements refresh (every 15 min, on a plan read) writes the free-plan flags back and the flip silently reverts; block it in dev by setting the Redis key `platform_plan:entitlements-refresh:<platformId>` with a long TTL. Then connect a **BYO key**, not the managed `ACTIVEPIECES` provider — per *Provider visibility* above, the managed row is hidden whenever `OPENROUTER_PROVISION_KEY` is unset, which is the normal local state, and a hidden provider makes `getChatProvider()` return null. Anything gated on a chat provider (chat itself, personalization research) stays silently disabled until a BYO row exists.
- **Since the AI steps moved to the worker, the server sees every step's model id, and `ai-execute-controller` is the one place to act on it.** The piece POSTs `/v1/ai/execute` with `provider` and `modelId`, the controller builds the `EXECUTE_AI` job from that body, and the worker runs whatever model id the job carries. So anything that needs to rewrite or validate the model a flow step is about to use belongs there, not in the piece — one server-side edit covers all five AI steps, and an old pinned piece bundle keeps working because it only forwards the string it stored. This is what makes storing a tier id in a step cheap: the bundle never has to understand it. Note the agent step does **not** come through here — it resolves separately over `getAgentConfig`, in `agent-config-rpc`. Two conditions sit under that. **An old bundle only forwards the string from `0.11.0` up** — `AI_PIECE_COST_BILLING_VERSION`, the first version whose `ai-step.ts` POSTs `v1/ai/execute`. A step pinned below it runs the pre-worker code that calls the provider from the sandbox, so a tier id there would reach OpenRouter as the literal string. That case is already closed: `migrate-v26-ai-piece-cost-billing` bumps every saved `piece-ai` step below `0.11.0` up to it, which is why a later migration can rewrite what a step stores without touching `pieceVersion` for the runtime. **And resolve on the API, never in the worker**: `ExecuteAiRequest.modelId` is a bare `z.string()` so a tier id passes validation, `getEffectiveProviderAndModel` and `aiUtils.buildWebSearchToolsOrThrow` both read `data.modelId` as a real model, and the `resolveAiProvider` RPC does not carry the model at all, so using it as the seam would mean widening the worker contract.
- **Managed billing can only price an OpenRouter call, and a call it cannot price is simply not billed.** `openRouterCostOf` (`server/utils/activepieces-ai-cost.ts`) reads `providerMetadata['openrouter'].usage.cost`, with one fallback that GETs `openrouter.ai/api/v1/generation?id=` and takes `total_cost`. If neither answers, `createBillingInfo` returns `undefined` and the turn is never reported — it runs, the customer keeps the output, and nothing is charged. Nothing logs "unbilled". This is what actually welds the managed provider to OpenRouter, more than the client does: `create-language-model.ts` shares one branch for `OPENROUTER` and `ACTIVEPIECES`, every tier `modelId` is an OpenRouter id, and `openrouter-api.ts` mints the per-platform key from `OPENROUTER_PROVISION_KEY`. So putting managed credits on any other gateway needs a second cost extractor and a second generation lookup **first**, or every call through it bills zero. For reference, no Vercel AI Gateway code exists here at all — the only gateway in the repo is Cloudflare's (`CLOUDFLARE_GATEWAY`, the `ai-gateway-provider` package), and `@ai-sdk/gateway` appears only as a transitive dep of `ai` that nothing imports. A BYOK user can already reach Vercel's gateway today through `AIProviderName.CUSTOM`, which takes a base URL plus hand-typed model ids.
- **A tier id that no longer exists resolves to the default tier of its surface, on every door, with a warning.** `resolveNamedModelId` (the path FLOW_STEP and AGENT take) tells a tier id from a model id by the slash rule: a bare slug is a tier id and goes through `resolveTier`, which returns the surface default for an id the file no longer carries and logs `Tier is no longer published; running the surface default` with the tier id and surface; a slashed id is a managed model id and is still refused with `The model "X" is not available on Activepieces AI credits` unless the release ships it or a published tier on that surface carries it. Every other source goes through `resolveModelIdForProvider` → `resolveTier`, the same default. So removing a tier from the file never fails a flow or an agent; it moves them onto the default until they are re-picked. The console may keep saying that removing a tier is safe. Decided 2026-09-26 with PR 2 of the configurable tiers work.
- **The console's Publish writes the live key directly; there is no draft, no staging and no filter.** `ai-pricing-service.ts` in `ap-analytics` writes `LIVE_PRICING_KEY` on Publish, and its "add tier" row seeds `id: tier-N`, `label: New tier`, so a half-finished experiment is one click from every server's cached tier list, with only the CDN cache plus the 15-minute in-process TTL in between (see the next bullet: the CDN part can be days, not minutes). Test against a local file with `AP_MODEL_TIERS_URL` (serve it on `127.0.0.1` and add `127.0.0.1` to `AP_SSRF_ALLOW_LIST`, which takes IPs or CIDRs, not hostnames, so `localhost` is still refused), never by publishing.
- **A console Publish can take days to reach servers: the plain `ai/pricing.json` URL is cached far longer than the `max-age=300` the console sets.** On 2026-10-15 origin had v7 (published 10:00 UTC) while `https://cdn.activepieces.com/ai/pricing.json` still served v6 with `cache-control: max-age=604800` and `age` ≈ 28h. The cache is the **DigitalOcean Spaces CDN** (`x-do-cdn-uuid` header), not Cloudflare (`cf-cache-status: DYNAMIC`): its edge cache TTL is 7 days and it overrides the object's own `Cache-Control`. `fetchTiers` fetches the plain URL, so every server keeps refreshing the stale copy. To verify a publish, compare `curl -s https://cdn.activepieces.com/ai/pricing.json | jq .version` with the same URL plus `?cb=$(date +%s)`. To see what a cloud server actually serves, look at `GET /api/v1/ai-providers/tiers` in the browser Network tab. If the versions differ, purge that key (`doctl compute cdn flush 046c3101-e4a3-49bd-a6f7-3dcd29a2a3d5 --files ai/pricing.json`, or Spaces → `activepieces-cdn` → Settings → CDN → Purge cache); re-publishing does not help.
- **A console older than the two-list change, or a rollback to one, strips the chat keys on read, and its next Publish drops them with no diff line.** The console parses the live file with a zod object, and zod strips keys it does not declare, so `chatTiers`, `chatDefaultTierId` and `chatFastTierId` vanish from the draft the moment an old console reads it. Publish writes the draft back, and the history diff shows nothing because that console never knew the keys. Activepieces then serves chat from the flow list again, silently, since the chat keys are optional. After any console rollback, republish from a current build and check `curl -s https://cdn.activepieces.com/ai/pricing.json | jq keys` lists all nine keys.
- **A managed text step stores a tier id from AI piece 0.12.0 on, and every managed model id stored before that came from a tier pick.** For the managed provider and a text step, `props.ts` (`packages/pieces/community/ai/src/lib/common/props.ts`) reads `GET /v1/ai-providers/tiers` with the engine token and offers the `flow` list as `{ label, value: tier.id }`. On any failure (a server without the route, a network error, an empty list) it offers the release's `ACTIVEPIECES_CHAT_TIERS`, still as tier ids, so a step saved on 0.12.0 always holds a tier id. `/models` is still called for managed image steps (`managedImageModelLabel` filters to `ACTIVEPIECES_IMAGE_TIERS`) and for own-key providers. Up to 0.11.0 the dropdown filtered `/models` to the bundled tier models and stored the concrete id, with no way to pin a model deliberately. So the migration from stored model ids to tier ids (ENG-753) has a trivially complete mapping: each stored id is one of the three bundled tier model ids of the release it was saved on. Until it runs, those steps follow the release, not the file.
- **A managed text step on 0.12.0 that still holds a concrete model id shows 'Select AI Model' and keeps running that model.** The options are tier ids now. `searchable-select.tsx` renders the placeholder when the value matches no option and keeps the value, and `dynamic-dropdown-piece-property.tsx` only clears it when a refresher (here `provider`) changes. The run is unaffected, because the slash rule passes a concrete id through. It only happens when something bumps `pieceVersion` in place, as a flow migration does. Options are fetched against the step's saved exact version, so a step left on 0.11.0 keeps its old options, and the update dialog's minor bump resets every input (`getInputAfterVersionChange`, `MINOR_OR_MAJOR`), so the user re-picks the model with everything else.
- **The AI step resolver skips image steps, because image tiers reuse the text tier ids.** `ACTIVEPIECES_IMAGE_TIERS` are `fast` / `smart` / `premium` too, and `aiJobFor`'s shared `modelId` feeds all five actions. Resolving a `GENERATE_IMAGE` job against the flow list would hand an image step a text model. The controller gates on `body.action`; any future image-tier resolution needs its own list.
- **`AIProviderModel` is declared twice, and a field added to one half is stripped on the wire.** There is one in `core/piece-types` (what pieces import, since they cannot import `@activepieces/shared`) and one in `core/shared` (what the server and the response schema use). They are not re-exports of each other — today the shared copy carries `metadata` and the piece-types copy does not. Fastify serialises the response against the *shared* schema and drops every key it does not declare, so adding a field to the piece-types copy alone type-checks in the piece, type-checks in the server, and arrives as `undefined`. A published tier label was nearly shipped that way. Any new field on a provider model needs both files.
- **No image model belongs to a chat tier, so a tier-based filter empties the image picker.** The managed provider is OpenRouter underneath and `openrouter-provider.listModels` tags a model `IMAGE` when its `architecture.output_modalities` says so — `ACTIVEPIECES` is deliberately absent from `NO_IMAGE_GENERATION_PROVIDERS`, and `google/gemini-2.5-flash-image` and friends are real managed models. But `ACTIVEPIECES_CHAT_TIERS` is three *chat* tiers, so no image model can ever match one. Any filter of the form "a managed model must belong to a tier" therefore drops every image model silently, leaving `aiProps({ modelType: 'image' })` with an empty dropdown and no error. Gate that filter on `modelType === 'text'`.
- **The api test suite runs on the bundled tiers, never the live file.** `packages/server/api/vitest.setup.ts` points `AP_MODEL_TIERS_URL` at `model-tiers.invalid`, so boot warm-up fails fast and every test resolves against `ACTIVEPIECES_CHAT_TIERS`. CI runners have egress, and the first time a resolver read the file in CI the live `defaultTierId` was `fast`, which turned a test asserting the release default (Expert) red with no code change behind it. A test that wants the published shape mocks `modelTierCatalog` (`test/unit/app/ee/agent/model-tier-fixture.ts`); an integration test that boots the app cannot be handed a file.
- **A published tier without `nativeModelId` keeps every BYOK key on the release's model for that tier id.** `nativeModelIdFor` (`ee/agent/agent-model-resolution.ts`) takes the tier's own `nativeModelId`, else the `nativeModelId` of the bundled `ACTIVEPIECES_CHAT_TIERS` entry with the same id, else null, which sends a curated provider to its first curated model and a catalogue-less provider to the first text model its key lists. The fallback is keyed by tier id only, so two lists sharing an id resolve to the same native model on every own-key provider, whatever their managed models are. The BYOK tier map (PR 6 of the configurable tiers work) is the real fix.
- **`premium` pairs `modelId: anthropic/claude-opus-4.8` with `nativeModelId: claude-opus-4-7` on purpose.** The managed tier runs Opus 4.8 through OpenRouter; a customer's own Anthropic key runs Opus 4.7. The reason is cost on the customer's key: Heavy on a BYOK key must not silently move the customer onto the most expensive model without them choosing it. The console's `defaults.ts` carries the same pair. Do not "fix" either copy.
- **Whatever the console publishes is in every Cloud customer's picker within about 20 minutes on a fresh page load, and about 35 minutes on a page left open: no draft, no staging, no filter.** `GET /v1/ai-providers/tiers` (`ListModelTiers` in `ai-provider-controller.ts`) returns `modelTierCatalog.current(surface)` for both lists, keyed `flow` and `chat`, to USER and ENGINE principals with no platform or project scope, because the tiers are global; the response schema puts only `id`, `label` and `modelId` on the wire. The web reads it through one hook, `aiProviderQueries.useModelTiers(surface)` (`features/platform-admin/hooks/ai-provider-hooks.ts`): the chat selector passes `chat`, the agent picker passes `flow`, and nothing but a test checks which one a caller passed, since the surface argument the server requires does not reach this layer. Both pickers show the release's bundled tiers while the call is in flight or has failed, so they never render empty. The web query is stale after 15 minutes and refetches on that interval while the tab is visible (`staleTime` alone only gates the mount, focus and reconnect triggers, so a tab left open would never refetch), which is where the two bounds come from: CDN `max-age` 5 min plus the server's 15-min TTL for a fresh load, plus one 15-min client tick for an open page, and the server serves its old copy once while it refreshes. The AI piece's own model dropdown reads the same route from piece 0.12.0 (the props.ts bullet above).

- **Every provider auth schema is written twice, and only one copy is the one the server reads.** `packages/core/utils/src/lib/ai-provider-credentials.ts` (plain zod) backs the API and the web app; `packages/core/piece-types/src/lib/ai-providers.ts` (`zod/mini`, so `.check(z.minLength(1))` and `z.optional(x)` instead of `.min(1)` / `.optional()`) is what `@activepieces/pieces-framework` re-exports to pieces and the engine. Adding a credential field to only one leaves the other silently stripping it — zod objects drop unknown keys, so the upsert still returns 200 and the failure surfaces later as a provider-side auth error. Change both.
- **A schema change in `packages/core/utils` is invisible to another package's tests until you build it.** Sibling packages resolve `@activepieces/core-utils` through its `main` (`dist/src/index.js`), so vitest in `core/ai-providers` keeps parsing against the stale build and a new optional field reads back as dropped. `cd packages/core/utils && npm run build` first. The API dev server is the opposite: `serve` runs under tsx with the tsconfig path alias, so it reads `src` directly — but it only watches `packages/core/shared/src/**`, not `core/utils`, so a core-utils-only edit needs a restart to take effect. The same staleness makes a **branch** look green when it is not: `dist` is gitignored, so after `git checkout` it still holds whatever the previous branch built, and a test that depends on a field added by a sibling PR passes locally and fails in CI. Rebuild `core/utils` after every branch switch before trusting a run.

- **`getEffectiveProviderAndModel` exists for one provider, and skipping it makes Cloudflare Gateway fall off the end of any per-provider switch.** It looks like generic provider resolution; it is not. It returns its input untouched for everything except `CLOUDFLARE_GATEWAY`, where it splits the model id and maps the submodel prefix (`anthropic`, `openai`, `google-ai-studio`, `google-vertex-ai`) back to the real provider. So a gateway model of `anthropic/claude-…` only reaches an Anthropic-keyed branch if you called it first. Any table or lookup keyed on `AIProviderName` — the web-search tool table, capability checks — has no `CLOUDFLARE_GATEWAY` entry and returns nothing for one that skipped the call: no tool, no error, no log. The AI piece calls it at the top of `buildWebSearchConfig`; server-side code that dispatches per provider needs the same call or the same resolution built in.

- **The Agent and the Direct AI steps disagree about whether OpenAI does web search, and both are right.** `WEB_SEARCH_MODE_BY_PROVIDER` has no `OPENAI` entry, so `supportsWebSearch(OPENAI)` is `false` — the Agent asks that question before offering the tool and therefore has never had OpenAI web search (it needed the Responses API, which agents did not use until GIT-1848; every OpenAI call is on Responses now, so adding `OPENAI: 'native'` is the open follow-up). Ask AI never asks it; it builds the tool straight from its own props and has had OpenAI web search all along. So `supportsWebSearch` is the *agent's* policy, not a statement of fact about the provider — reusing it as a general gate silently removes a capability Ask AI users have today. Whoever unifies the two paths has to keep the OpenAI branch reachable without routing it through that gate. The two paths also disagree about the unsupported tail: the piece's `webSearch` prop is a plain checkbox with no provider gating, and `createWebSearchTool` ends in `default: throw new Error('Provider X is not supported for web search')` — so ticking it on Azure, Mistral, xAI, or a Cloudflare Gateway model whose submodel prefix is not one of the four in `CF_GATEWAY_SUBMODEL_TO_PROVIDER` **fails the step**. The server's `buildWebSearchTools` returns `{}` for the same input, because the agent gates before calling it. The builder cannot simply be made to throw: `supportsWebSearch` is true for the `plugin` providers (`openrouter`, `activepieces`), the agent calls the builder for them, and there is no native entry for either because the plugin rides on the model settings instead — so `{}` is the correct answer there. The piece has the same shape, returning early for OpenRouter before the switch that throws. Hence two entry points: `buildWebSearchTools` stays total for callers that gate first, `buildWebSearchToolsOrThrow` reproduces the piece's error for callers that do not.

- **The AI-provider create/update endpoints reject unknown `auth` keys rather than stripping them** (`strictAuth` in `packages/core/shared/src/lib/management/ai-providers/index.ts`). This is deliberately the request boundary only: `aiProviderCredentials()` still parses *stored* rows through the loose schemas, because a strict parse of a legacy row falls through to the `auth: {}` fallback and silently empties a working key. If you tighten the shared schemas themselves, you move a loud 400 into a quiet credential loss.
- **`aiProviderModule` is registered unconditionally, `agentModule` is not.** In `app.ts`, `aiProviderModule` sits outside the edition switch, while `agentModule` appears only in the `CLOUD` and `ENTERPRISE` arms — so an AI route hung off the provider module is on Community for free, and the same route hung off the agent module 404s there. That asymmetry is the cheapest way to satisfy [000036](../decisions/000036-the-worker-side-ai-path-is-community-edition-core.md): pick the module and there is no edition branch left to get wrong.
- **Chat falls back to the managed Activepieces row, so "no key is flagged for chat" is no longer a wall.** `pickChatRow` takes the `enabledForChat` row if there is one and otherwise the visible `ACTIVEPIECES` row; `findAvailableChatProviderRow` and both list endpoints share it, so what the picker shows is what a turn will actually run on. Before that, a platform whose managed row sat at `enabledForChat = false` — and nothing else flagged — got `SetupRequiredState` ("Set up an AI provider to get started") a second after the chat loaded. Migration `1802…AddChatRolloutFreeCreditGrant` was meant to repair exactly that state and matched zero rows: it compared `"provider" = 'ACTIVEPIECES'` while `AIProviderName` values are stored lowercase (`'activepieces'`). Enum values in raw SQL are the lowercase string, never the TS member name — grep the enum before writing the WHERE clause.

- **Multi-key resolution is deterministic, not configurable.** When several keys of one provider are eligible for a project, `resolveEligibleRow` picks by most specific `projectScope` (`selected` > `except` > `all`), newest `created` breaking ties — there is no priority/default field (decision: [providers-redesign-before-routing](../decisions/providers-redesign-before-routing.md)). The ACTIVEPIECES provider stays a singleton — `create()` rejects it (`aiProvider.activepiecesIsManaged`). That ranking is the *fallback*: a step or agent may also pin a key outright (decision: [000030](../decisions/000030-a-step-may-pin-an-ai-provider-key-and-omitting-one-can-only-narrow.md)), in which case `resolveRowForScope` serves that row after checking it is eligible for the caller's project.
- **Every resolver takes a required `ProviderScope`; there is no "no project" default** (decision: [000027](../decisions/000027-ai-provider-resolution-takes-a-required-scope-and-splits-reads-by-trust-level.md)). `getConfigOrThrow` / `getChatProvider` / `getChatProviderName` / `listModels` all take `scope: { type: 'project', projectId } | { type: 'platform' }`. This is deliberate and load-bearing: the first cut made `projectId` optional and treated its absence as "every key is eligible", so each new call site that forgot to thread it silently bypassed project scoping — the agent piece/knowledge-base tool handlers, the chat model picker, and the `configId` model lookup each reopened the same hole in turn. Making the argument required turns an omission into a compile error, and `{ type: 'platform' }` at a call site is a reviewable claim rather than an accident. Only three consumers are legitimately platform-wide: the tool-search embedder, chat memory extraction, and the managed-ACTIVEPIECES singleton.
- **The last fail-open scope lived in the helper that *builds* the scope, not in the resolvers.** Chat resolves the project a turn runs in (`selectRunProject`) before it resolves a credential, and that project is nullable — a user who can no longer see any project gets `null`. The first cut turned `null` into `{ type: 'platform' }`, which hands the run every key on the platform and is exactly the hole a required `ProviderScope` was meant to close. `agentHelpers.runScopeOrThrow` refuses the run instead; the analytics and billing paths, which want the provider's *name* and not a credential, take the nullable project id themselves (`resolveChatProviderName`) and report no provider for a conversation that has none. Rule of thumb: a nullable id feeding a scope constructor is the shape to look for, not a missing argument.
- **The managed ACTIVEPIECES row is a singleton the database enforces, not the code.** `create()` rejecting it only covers the admin route; the row is *also* auto-provisioned from `listVisibleRows` on every list, so two concurrent `GET /v1/ai-providers` both missed the `existsBy` and inserted once the unique `(platformId, provider)` index was dropped. `idx_ai_provider_platform_id_managed` — unique on `(platformId)` `WHERE provider = 'activepieces'` — keeps it single, and the insert carries `ON CONFLICT DO NOTHING` so the losing racer is a no-op rather than a 500. Preferred over a distributed lock because the invariant holds even for a writer that never takes one.
- **Reads are split by trust level, and mixing them back together is how the scope bypass keeps returning.** Runtime/project reads are `GET /v1/ai-providers?projectId=` (deduped to one entry per provider, `{provider, name, enabledForChat}` only) and `GET /v1/ai-providers/:provider/models?projectId=` — both `securityAccess.project([USER, ENGINE], undefined, QUERY)`, so an ENGINE principal supplies its own project and a USER must name a project it belongs to, and both always apply the resolved key's `modelScope` allow-list. Admin reads are `GET /v1/ai-providers/configs` and `GET /v1/ai-providers/configs/:id/models` — `platformAdminOnly`, addressing an exact row and returning the *unfiltered* model list because that is what an admin picks the allow-list from. Never widen the project routes to accept a config id, and never hand `projectIds`/`modelIds` to a project caller: those are other projects' identifiers. That rule is pinned by a guard test asserting the *exact* key set of a project entry (and of each item in its `keys` array), so adding a field to the project-facing response is meant to fail `ai-provider.test.ts` until someone states the field is safe to expose.
- **Two keys may legitimately hold the same secret, and you could not detect it anyway.** One API key scoped to two rows with different `modelScope`/`projectScope` allow-lists is a supported setup, not a mistake. Blocking it is also impractical: `encryptUtils.encryptObject` uses a fresh random IV per write, so the same credential stores as different ciphertext every time — deduping would need a separate HMAC column. Key *names* are the thing worth constraining, since a picker showing two rows called "Anthropic key" is unpickable.
- **`mockAndSaveAIProvider` uses `save`, not upsert** — the old `(platformId, provider)` ON CONFLICT target died with the unique index; seeding the same provider twice now creates two keys, which is usually what a test wants.
- ACTIVEPIECES auto-provision needs `OPENROUTER_PROVISION_KEY` env var set AND `aiCreditsEnabled` true.
- **`secret: true` on a credential field is not about masking, it is what stops a stored credential being prefilled back into the form.** In `PROVIDER_CREDENTIAL_FIELDS` the flag feeds `secretKeysOf`, and the dialog's default-values builder blanks exactly those keys when reopening an existing key; the password-style `Input` is a side effect of the same flag, not its purpose. So dropping it to change how a field *renders* silently turns a stored secret into a prefilled value. Render differently instead: `type` (`dictionary`, `textarea`) is checked ahead of `secret` in `CredentialFieldInput`, so a field can keep the flag and still draw as something other than a masked input — that is how the Vertex service-account JSON gets a monospace textarea while staying blanked on edit. Both the initialiser and the required-field check filter only `dictionary`, so any other `type` is initialised and validated like a normal credential.
- **Custom sends the API key exactly as typed; LiteLLM adds `Bearer`.** Custom puts `[apiKeyHeader]: apiKey` on the wire raw, both in `createLanguageModel` and in `suggestModels`, so a gateway that wants `Authorization: Bearer …` needs the admin to type `Bearer ` into the key. LiteLLM goes through `litellmAuth.buildHeaders`, which adds the prefix for `Authorization`. The connect dialog's header help text differs per provider for this reason; keep them apart if you share field definitions.
- **Adding a provider is a leaf change, and the credential fields are the only part that is not.** A new vendor touches six places: the `AIProviderName` enum (`packages/core/utils/.../permission.ts`), its auth/config schemas plus the two unions and `ProviderConfigUnion` in `packages/core/shared/.../management/ai-providers/index.ts` (all in the per-provider region, well above the generic request/response schemas at the bottom), a strategy file registered in `ai/providers/index.ts`, the model factory switch, name/logo/markdown in `packages/web/src/features/agents/ai-providers.ts`, and translation keys. None of that is the credential form: extra fields beyond `apiKey` (Azure's `resourceName`, Bedrock's region) are declared in **one** file — `PROVIDER_CREDENTIAL_FIELDS` in `.../setup/ai/providers-tab/provider-credentials.ts`, which falls back to `DEFAULT_CREDENTIAL_FIELDS` (a single `apiKey`) for any provider with no entry, so a plain API-key vendor needs no UI work at all. That file replaced the deleted `universal-pieces/upsert-provider-config-form.tsx` in the multi-key redesign, so a provider authored against an older branch loses its custom fields on merge **silently** — git resolves delete-vs-modify by taking the deletion, no conflict marker, and the provider just becomes unconfigurable in the admin UI. A vendor with no `/models` endpoint also belongs in `MANUAL_MODEL_PROVIDERS` in that same file (`CUSTOM`, `CLOUDFLARE_GATEWAY`) so the admin enters model ids by hand. Everything else about a provider is orthogonal to multi-key: that is a table-level change (drop `UNIQUE (platformId, provider)`, add the four scope columns, keep a unique partial index for `activepieces` only), so a provider inherits multi-key with no provider-side code, and the admin providers-tab enumerates `SUPPORTED_AI_PROVIDERS` from `packages/web/src/features/agents/ai-providers.ts` rather than a catalog of its own, so a new vendor appears there on its own. When basing provider work off a branch that predates the redesign, re-check `provider-credentials.ts` after the merge.
- **The OpenAI-compatible vendors (xAI, DeepSeek, Z.ai, Qwen, MiniMax, Moonshot) share one strategy rather than a file each**, via `openAiCompatibleVendor({ name, provider })` in `ai/providers/`, with defaults in `OPENAI_COMPATIBLE_VENDOR_BASE_URLS` and an optional per-key `baseUrl` override because four of them run separate China and international endpoints. Their `listModels` GETs `{baseUrl}/models`, which the vendor docs mostly do not document — **confirmed working against live keys for DeepSeek, Z.ai, MiniMax and Moonshot** (Qwen still unverified), so don't redo that research. If a future vendor turns out to lack `/models`, the fallback is the manual-models path rather than a bespoke strategy. Unlike its siblings this factory uses `safeHttp.axios`, not `httpClient` from `pieces-common`: the base URL is admin-supplied, so it must go through the SSRF filter.
- **Adding an API-key vendor touches these files, and two misses only fail at runtime.** `AIProviderName` (`core/utils/.../permission.ts`); the `AiProviderCredentials` union (`core/utils/.../ai-provider-credentials.ts`) — miss it and `aiProviderCredentials()` throws at run time, because `createLanguageModel`'s `never` check cannot see a provider the union never admits; `ProviderConfigUnion` (`core/shared/.../management/ai-providers`) — miss it and creating a key returns 400; `OpenAiCompatibleVendor`, `OPENAI_COMPATIBLE_VENDOR_BASE_URLS`, `NO_IMAGE_GENERATION_PROVIDERS` and the exhaustive `AI_PROVIDER_CAPABILITIES` (`core/piece-types/.../ai-providers.ts`); the `createLanguageModel` switch; the exhaustive `aiProviders` registry (`server/api/.../ai/providers`); web `SUPPORTED_AI_PROVIDERS`, the API-key group in `provider-request.ts`, and the markdown translation key; `MODELS_DEV_PROVIDER` in `tools/scripts/sync-model-catalog.ts`; the two provider lists in `docs/admin-guide/guides/setup-ai-providers.mdx`; a slot in `PROVIDERS_BY_IMPORTANCE` (`setup/ai/providers-tab/index.tsx`), which orders the Providers page and otherwise drops the new vendor after Other (OpenAI Compatible); and a CDN logo. The AI piece no longer has its own factory: `ai-sdk.ts` / `buildLanguageModel` were deleted in #15495 and AI steps run in the worker through `createLanguageModel`. `NO_IMAGE_GENERATION_PROVIDERS` is still load-bearing for text-only vendors, because `createImageModel` returns `undefined` for them and that list is what turns an image request into a clear error. A new `AIProviderName` also needs an AI piece version bump, because `toProviderName` (`pieces/community/ai/.../common/props.ts`) checks the enum bundled with the published piece and drops a value it does not know. Core libraries resolve each other from `dist`, so rebuild `core-utils` and `core-piece-types` (`npm run build` in each) before running a dependent package's tests, or the new enum value reads as `undefined`.
- **A provider missing from `SUPPORTED_AI_PROVIDERS` is invisible on the Providers page, but its keys still work.** `ProviderGroup` returns `null` for it, so an API-created key can't be seen, edited or deleted in the UI, yet it feeds the model picker and the default chat model fallback ("Not set · uses <its model>"). That is how a provider ships hidden before its form (LiteLLM PRs 1–4). Separately, on CE the AI Center sidebar has no sub-items: Capabilities and Usage need EE or Cloud, Tiers needs Cloud.
- **Overriding `fetch` on a provider silently switches off key-health reporting.** `...observed` *is* a `fetch` (`observedProviderFetch`), so any later `fetch` in the same options object replaces it rather than adding to it — spread order decides, and nothing errors or logs. A path that needs its own `fetch` (the Responses header-stripping one, the Cloudflare gateway's `Authorization`-deleting one) must take `observedProviderFetch(options.onOutcome)` as a delegate and call it, not call `globalThis.fetch`. Resolve the delegate at call time rather than capturing it at model-construction time; late binding is both more correct and the only way a test can stub the global. This is the same family as the merge hazard above — the failure is always one provider, or one path within a provider, quietly not reporting.
- **A provider branch merges cleanly and still opts out of whatever the switch gained meanwhile.** `createLanguageModel` applies cross-cutting behaviour by spreading it into *every* case — `...observed` (`observedProviderFetch`, added by the key-health work) is the current one. Git treats a long-lived branch's new `case` as an added block, so it lands with none of the spreads its siblings picked up, and the merge reports no conflict: the new provider is simply the one that never reports health. Nothing fails, nothing logs. After merging main into any provider branch, diff your `case` against a neighbouring one and check the spreads match, rather than trusting a clean merge.
- **Only two `Record<AIProviderName, …>` maps are exhaustive, so the enum addition barely fights back.** `AI_PROVIDER_CAPABILITIES` (piece-types) and `aiProviders` (`ai/providers/index.ts`) must gain an entry; `PROVIDER_CREDENTIAL_FIELDS`, `ALLOWED_CHAT_MODELS_BY_PROVIDER`, `PROVIDER_MAX_CONTEXT_TOKENS`, `EMBEDDING_MODELS` and `WEB_SEARCH_MODE_BY_PROVIDER` are all `Partial<Record<…>>` and degrade silently to defaults. `grep -rn 'Record<AIProviderName' | grep -v 'Partial<'` is the whole compile-time safety net — every other consequence of a new provider is a runtime question you have to reason about yourself.
- **The repo runs three `@ai-sdk` generations at once, and the vendor package version is what pins you to one.** `@activepieces/ai-providers` is on ai@7 (`@ai-sdk/google` 4.0.29, `openai-compatible` 3.0.18, provider-utils 5.0.16); the AI piece is on ai@6 (google 3.0.65, openai-compatible 2.0.42, provider-utils 4.0.24); older code sits on the ai@5 line (google 2.0.71, provider-utils 3.0.24). A vendor SDK therefore needs a *different release per factory* — `@ai-sdk/google-vertex` is 5.0.36 for ai@7 and 4.0.113 for ai@6 — and taking the newest for both silently mixes `LanguageModelV3`/`V4` and fails at the engine boundary. Match transitive deps rather than guessing: `curl -s https://registry.npmjs.org/@ai-sdk/<pkg>` and pick the version whose `dependencies` equal the target package's own pins. Faster still, `ls node_modules/.bun/ | grep '<pkg>@'` usually already lists every generation bun has resolved.
- **The OpenAI-compatible provider can speak the Responses API, and `@ai-sdk/openai-compatible` is not how you get there.** That SDK exposes only `chatModel` (`/chat/completions`) and `completionModel` — no responses model in any version. The route is `@ai-sdk/openai`, which accepts a custom `baseURL` + `headers` and exposes `.responses()`; the Cloudflare gateway's openai branch already did this before the `apiStyle: 'chat' | 'responses'` config field existed. Header precedence is the subtlety: the SDK builds `{ Authorization: Bearer <apiKey>, ...options.headers }`, so a caller header wins on `Authorization` (the Bedrock case, where the key *is* the bearer token) but a custom `apiKeyHeader` such as `x-api-key` leaves the SDK's default `Authorization` riding along. `withUserAgentSuffix` drops `undefined` values, but `OpenAIProviderSettings.headers` is typed `Record<string, string>`, so the way to strip it is the header-stripping `fetch` the Cloudflare branch uses — no cast needed, and worth reaching for straight away rather than writing the duplicate header up as an accepted limitation. Strip only when `apiKeyHeader` is not itself Authorization, compared **case-insensitively**: HTTP header names are, and an admin can type `authorization`. Confirmed shape, since it is easy to assert on the wrong key: the SDK lowercases everything, so a mis-set provider really does emit `{authorization: 'Bearer <key>', 'x-api-key': '<key>'}` — the credential twice.
- **AWS's OpenAI-compatible surface is two endpoints, and "Bedrock Mantle" is a real AWS product name, not a customer's nickname.** `bedrock-runtime.{region}.amazonaws.com/openai/v1` (AWS-recommended) and `bedrock-mantle.{region}.api.aws/v1` both serve Chat Completions *and* Responses, both authenticate with a Bedrock API key as `Authorization: Bearer`, and both take an unmodified OpenAI SDK. Mantle alone adds server-side tools and web search, `background=true` async inference, and Projects/Workspaces; bedrock-runtime alone has Guardrails, cross-Region inference and intelligent prompt routing. So a ticket claiming "Mantle only supports responses" is half right — it serves chat/completions too, and the real loss is the Responses-only capabilities.
- **Vertex is a different door onto the same models, not a different vendor.** The `GOOGLE` provider is the Gemini Developer API (`generativelanguage.googleapis.com`, static API key); `VERTEX` is Vertex AI (`{region}-aiplatform.googleapis.com/v1beta1/projects/{project}/locations/{region}/publishers/google`, service-account OAuth2 with ~1h tokens) — the same Gemini models, but billed to the customer's GCP account and inheriting IAM, VPC-SC, CMEK, data residency and audit logs, plus Model Garden's Claude/Llama/Mistral. That rotating token is why the CUSTOM provider can never reach Vertex: `buildOpenAICompatibleHeaders` injects one static header, which is why the workaround used to be a LiteLLM proxy. `@ai-sdk/google-vertex` does the JWT-to-token exchange itself, so the native provider needs no proxy. Vertex models are entered by hand (`MANUAL_MODEL_PROVIDERS`) on purpose: availability is project-, region- and Model-Garden-specific, and a `publishers/google` listing would miss exactly the third-party models that motivate choosing Vertex. **But "same models" only holds for Gemini.** Model Garden also serves Anthropic, Meta, Mistral and xAI, and those do not share Gemini's API surface — `@ai-sdk/google-vertex` ships separate `/anthropic`, `/maas` and `/xai` entry points for exactly that reason, so one `createVertex(...)` call does not cover Vertex. Because models are typed in by hand, a Claude id reaches the factory as an ordinary string and, unrouted, gets built with the Gemini client and fails at the endpoint. `create-language-model.ts` routes ids containing `claude` to `createVertexAnthropic`; `vertexClientFor` picks the client from the id shape, so all three are routed. Because the shapes are unambiguous the check is a lookup, not a tuned heuristic — and note the publisher prefix has to be tested *before* the `claude` one, or a MaaS path containing "claude" routes to the Anthropic client. Routing is cheap because the id shapes are distinct, not because a heuristic was tuned: `gemini-2.5-pro` is bare, Anthropic carries an `@date` (`claude-3-5-sonnet@20241022`), and MaaS is publisher-prefixed and suffixed (`meta/llama-4-scout-17b-16e-instruct-maas`). The publisher prefix is a stronger discriminator than the `includes('claude')` match currently shipped, and it looks like an argument for listing models rather than taking them by hand — but it is not, see below: the listing is scoped to a single publisher, so Model Garden ids stay hand-typed and the string matching stays with them. **Listing was considered and rejected**, and the reason is worth keeping so it does not get reopened, and **the response shape does not need a GCP account to learn** — every Google API publishes an unauthenticated discovery document: `curl -s 'https://aiplatform.googleapis.com/$discovery/rest?version=v1beta1'` returns ~5MB of JSON whose `schemas` are the authoritative request/response types. `publishers.models.list` returns `{ publisherModels: PublisherModel[], nextPageToken }`, and `PublisherModel` is `{ name: 'publishers/google/models/<id>', versionId, versionState (STABLE|UNSTABLE), launchStage (GA|PUBLIC_PREVIEW|PRIVATE_PREVIEW|EXPERIMENTAL), openSourceCategory, supportedActions, predictSchemata, frameworks, parent }`. **There is no `displayName` on it** — the one in `google-vertexai`'s piece comes from `@google/genai`'s `Model` type, which is a different shape, so copying that code gives every row a blank label with nothing failing. `displayName` exists only on the nested `parent`, which is the base model a tuned model derives from. The decisive fact is in that same doc: `publishers.models.list` takes `parent` matching `^publishers/[^/]+$` — **one publisher per call**. Listing everything therefore means hardcoding the set of publishers to enumerate, which is the same shape of hardcoding it was meant to remove, and Model Garden models would still need hand entry, leaving two mechanisms where there is now one. `VERTEX` stays in `MANUAL_MODEL_PROVIDERS` alongside `CUSTOM` and `CLOUDFLARE_GATEWAY`, which fits: Vertex model availability really is per-project. Reach for the discovery doc before a live probe on any `*.googleapis.com` integration; a curl against the real endpoint needs **billing attached to the project even for a read** (`403 BILLING_DISABLED`), so it is the slower path and it is not free to set up. The `x-goog-user-project` header is not optional when probing with a personal login: without it aiplatform answers `403 SERVICE_DISABLED` naming `consumer: projects/32555940559`, which is Google's shared gcloud CLI project rather than yours, and the message talks about quota projects rather than the missing header. This is a user-credential quirk only — a service account carries its own `project_id`, so the provider itself never hits it. **The multi-vendor story is text only.** Vertex's image models are Google's own Imagen (`imagen-3.0-*`, `imagen-4.0-*`), not Model Garden's third parties, and `createVertex(...)` also exposes `imageModel`, `video`, `speech`, `transcription` and `textEmbeddingModel` — none of which we wire today. Imagen is wired (`createVertex(...).imageModel()`, with a `VERTEX` case in the piece's image switch, so `VERTEX` is no longer in `NO_IMAGE_GENERATION_PROVIDERS`); video, speech, transcription and embeddings are not. Embeddings still have no `EMBEDDING_MODELS` entry, so those paths fail with an explanatory message instead of reaching a switch that has no case for it.
- **Chat-tier resolution answers "which model" from three places, and only one of them is the key.** `resolveModelIdForProvider` (`ee/agent/agent-model-resolution.ts`, spread into `agentHelpers`) now prefers the resolved key's own configured text models when its config carries a `models` array — that is the `MANUAL_MODEL_PROVIDERS` case (Vertex, Custom, Cloudflare Gateway), where the admin typed the exact ids the key exposes. **An empty catalog is not a missing one**, and collapsing the two is how the first cut of this went wrong: a helper returning `string[] | undefined` treated "lists zero text models" the same as "has no catalog", so a key configured with only image models fell through to the curated list and resolved to a Gemini id it never offered. An admin-listed catalog is the whole truth about a key, so an empty one now refuses the turn with a message rather than guessing. Otherwise it falls back to the static `ALLOWED_CHAT_MODELS_BY_PROVIDER`, and if the provider is missing from that too it returns the raw tier id with its vendor prefix stripped — which is how a Vertex key once resolved to `claude-sonnet-4-6` and handed it to the Gemini client. That map is `Partial<`, so omitting a chat-capable provider compiles cleanly and fails only at the endpoint. **The key's own `modelScope`/`modelIds` allow-list is applied last, to whichever candidate list was chosen**, and a resolution with nothing left refuses the turn rather than returning a model the key forbids. `GetProviderConfigResponse` carries both fields for that, populated at all three construction sites (`getChatProvider`, `getConfigOrThrow`, `enrichWithKeysIfNeeded`); adding fields there is backward compatible because the AI piece reads the response as a typed shape rather than parsing it. I first recorded this as too big for a provider PR — "a wire contract the engine and the AI piece both consume" — having never measured it. It was about thirty lines across four files. Measure before recording something as out of scope; the estimate outlives the guess. **The resolver is exported and has five call sites across four files, and only two of them are runs.** `agent-rpc-handlers` resolves the chat turn's model *and* its fast model, and `chat-personalization-service` resolves both for research — those four must be given the resolved row's `config`, `modelScope` and `modelIds`, or the key's constraints are bypassed on exactly the paths a user takes. `resolveTierModel` inside `agent-helpers` is the fifth. `agent-service` also calls it but holds only a provider name and sets a stored default on a draft, so it is correctly left thin (`agent-draft-ai` did too, until #15676 deleted it). Adding a parameter here means grepping `resolveModelIdForProvider|resolveFastModelId` across `packages/server`, not editing the call site in front of you — threading only `resolveTierModel` looks complete, passes every test, and still leaves the real chat path unconstrained. The `surface` argument was added as a required key on the whole family for exactly that reason: a missed call site is a compile error, except inside a `vi.mock` factory, which is not type-checked and has to be updated by hand. One trap when touching this function: a tier id (`fast`, `smart`) is **not** a model id, so the preferred pick must be the resolved native id unless the selection is itself among the candidates — passing `selectedModel` straight through makes `resolveFastModelId` return the first curated model instead of the fast tier's.
- **A failed credential validation tells the admin nothing, for every provider except Cloudflare Gateway.** `aiProviderService.validateProviderCredentials` gates the upstream message behind `includeHttpErrorInMessage`, which is `provider === CLOUDFLARE_GATEWAY` and nothing else, so everyone else gets a bare `Failed to validate credentials for <name>`. The cause is not lost — it is logged one line earlier (`log.error({ error }, '[aiProviderService#validateProviderCredentials] ...')`) and passed as the `httpErrorResponse` error param — but **web never renders `httpErrorResponse`**, so the only way to diagnose a rejected key is the server log. Grep the log for `validateProviderCredentials` before assuming the provider integration is broken — that text is the whole diagnosis, and it is often not about credentials at all. Confirmed case: a brand-new xAI team with no credits purchased answers `GET /v1/models` with `403 permission-denied — Your newly created team doesn't have any credits or licenses yet`, naming the console page that fixes it, and we render that as "Failed to validate credentials for xAI" — sending the admin off to regenerate a key that was never wrong. Vendors also phrase real key failures inconsistently (xAI uses `400 Incorrect API key provided`, not a 401). The corollary: **a provider that saves without error is not a working provider.** A no-credits 403 and a bad key are indistinguishable in the UI, so only an actual generation proves a key end to end. This is an admin-only surface (`platformAdminOnly`), so there is little reason to keep hiding it.
- **A provider's logo is an asset someone has to upload, not something the code ships.** `AiProviderInfo.logoUrl` in `packages/web/src/features/agents/ai-providers.ts` is a plain string rendered into an `<img src>`, and every provider points at `https://cdn.activepieces.com/pieces/<slug>.png` — nothing is bundled. Adding a provider therefore carries a cross-team dependency with no compile-time or test signal: a slug with no asset behind it renders a broken-image icon in the platform admin list, and only a live request tells you. Check the URL with `curl -o /dev/null -w '%{http_code}'` before assuming it works — a vendor that already ships as a *piece* usually has its logo there already (`deepseek.png`, `grok-xai.png` did), so start the upload request only for the genuinely missing ones. A Vite asset import also satisfies `logoUrl` (see `GoogleIcon` in `platform/security/sso/index.tsx`) and removes the runtime CDN dependency for air-gapped installs, but it diverges from every other provider — treat it as a fallback, not the default. Upload it trimmed, with no built-in whitespace and no solid background: `ProviderLogo` draws every logo at 22px inside a 36px `LogoPlate`, so padding baked into the PNG (Qwen, Azure) makes it look smaller than its neighbours and a filled square (xAI) looks heavier. No code fixes that. Use the symbol only, square, never a wordmark lockup: Meta's and NVIDIA's first uploads were full brand lockups and rendered as unreadable blobs. To cut one from a lockup: `magick in.png -crop WxH+X+Y +repage -fuzz 8% -transparent white -trim +repage -background none -gravity center -extent "%[fx:max(w,h)*1.08]x%[fx:max(w,h)*1.08]" -resize 256x256 out.png`.
- **The manual-models panel stores a model id and nothing else, so every model an admin adds is typed `TEXT`.** `MANUAL_MODEL_PROVIDERS` (Vertex, Custom, Cloudflare Gateway) persist a `ProviderModelConfig` of `{ modelId, modelName, modelType }`, but since #14825 the UI keeps only `string[]`: `ManualModelList` is a chip list of ids, and `config-detail.tsx` rebuilds the objects on save with `?? { modelId, modelName: modelId, modelType: TEXT }`. The old `universal-pieces/model-form-popover.tsx` had all three fields; replacing it with chips dropped two of them, and no schema or test noticed because the reconstruction is type-correct. The visible damage is not just a mislabelled model: `cloudflareGatewayProvider.validateConnection` pings **only text models** with a `/chat/completions` request, so one image model added through the UI is validated as chat, fails, and takes the whole key down with `These models have issues: ...` — the admin cannot save a key that has an image model on it at all. Anything reading `modelType` downstream (the agent text-model filter, `ap-list-ai-models`, the builder's image pickers) is wrong the same way. Fixed by holding `ProviderModelConfig[]` in the draft rather than `string[]` (no migration; stored rows always had the right shape), so the standing rule is that **a manual-models UI must carry `modelType` all the way to the payload** — a panel that collapses models to ids reintroduces exactly this, and nothing downstream will complain until an admin adds an image model. Keys saved while the bug was live still hold image models typed `TEXT`; the type badge on each chip flips one in place, which is why it is editable rather than display-only.
- **Custom keys get model suggestions; Vertex and Cloudflare Gateway stay typed by hand on purpose.** `GET /v1/ai-providers/configs/:id/suggested-models` calls the strategy's optional `suggestModels`, which only `openAICompatibleProvider` implements: `GET {baseUrl}/models` with the key's own header, through `fetchOpenAiModelIds` (shared with the vendor list), filtered by `isChatModelId`. The ids feed a `<datalist>` on the manual-models input, so free text still works and the admin still picks `modelType` (`/models` does not say text or image). Suggestions never touch `listModels`, which stays `config.models` because tiers and key scope read it. A failed fetch returns `[]` and logs, so a LAN or localhost base URL that `safeHttp` blocks (self-hosted Ollama, unless in `AP_SSRF_ALLOW_LIST`) just shows no suggestions. Vertex is not listed for the reason in the Vertex bullet (one publisher per call), and Cloudflare's gateway has no models endpoint. Any fetch to an admin-supplied URL needs three bounds, not one: axios's `timeout` only fires when the connection goes idle, so a server that drips bytes holds it open, and a huge body is read into memory in full. `fetchOpenAiModelIds` passes `timeout`, `signal: AbortSignal.timeout(...)` and `maxContentLength`. Trim a trailing slash with `/\/$/`, not `/\/+$/`: the `+` form is quadratic on a long run of slashes and `baseUrl` has no length limit (both from review on #16382).

- **`AIProviderConfig` is an *untagged* `z.union`, so a new provider's config schema must sit ahead of the empty ones — and "empty" includes a schema whose every field is optional.** Zod strips unknown keys and a union returns the first member that parses, so `AnthropicProviderConfig` (`z.object({})`) matches *any* object: list it before a `{ baseUrl?: string }` config and a configured base URL is silently reduced to `{}` — no error, no log, the admin's override just stops existing on the next read. The file carries an `Order matters` comment, but it says "empty ones last", which reads as though only a literal `z.object({})` is at risk. The rule is **not** "after the last schema with a required field" — that phrasing reads as "append after `BedrockProviderConfig`" and is how a Vertex config got silently reduced to `{ region }`, losing `project` and `models` with no error. What matters is the *subset relation*, not the count: `BedrockProviderConfig` is `{ region }`, a strict subset of `{ project, region, models }`, so it matches a Vertex object first and strips the rest. Order by specificity — any schema whose required keys are a subset of yours must sit **after** you. Check with a one-line parse before trusting the order: `AIProviderConfig.parse(yourConfig)` must return every field it was given. `ProviderConfigUnion` is discriminated on `provider` and so is immune; only the two untagged unions (`AIProviderConfig`, `AIProviderAuthConfig`) bite. Both live twice — `packages/core/shared/.../management/ai-providers/index.ts` (zod classic) and `packages/core/piece-types/.../ai-providers.ts` (zod/mini, the copy pieces use) — and every provider edit has to land in both. **The third copy moved rather than disappearing.** `.../setup/ai/universal-pieces/` is gone and the connect dialog now builds from `PROVIDER_CREDENTIAL_FIELDS`, but `config-detail.tsx` declares its own `ManualProviderConfig = z.union([...])` that gates whether the manual-models panel sends `config` at all. A provider missing from it fails `safeParse`, its models are dropped from the payload, and they silently vanish on reload — which for a `MANUAL_MODEL_PROVIDERS` member means the only way to choose a model does not work. So it is still **three** unions per provider: the two shared copies plus this one, and it is the easiest to miss because it lives in a component and nothing references the provider by name. The rest of this sentence describes that deleted file and is kept only as history:  `createFormSchema` in the admin dialog (`.../setup/ai/universal-pieces/upsert-provider-dialog.tsx`) re-declares a per-provider schema, branching explicitly on Azure / Cloudflare / Custom / Bedrock and falling through to a generic case whose `config` is a union of three empty objects. **A provider with a non-empty config and no branch there loses that config entirely** — `zodResolver` hands react-hook-form the *parsed* value, so the strip happens before submit and the setting is never sent, with no error anywhere. Fixing the shared union does not fix this one; grep for every union of config schemas when adding a provider. The dialog only diverges from the correct `ProviderConfigUnion` to make auth optional in edit mode, so collapsing it onto the shared discriminated union is the real repair. (Testing that file directly is awkward: importing it pulls in a transitive dep that touches `document` at import time, which a node-env vitest cannot load — the schema factory would have to move out of the component file first.)
- **Every catalog field is optional, and two providers never match at all.** Azure's `listModels`
  returns *deployment names* (arbitrary admin-chosen strings), and CUSTOM / CLOUDFLARE_GATEWAY ids are
  hand-typed, so `modelCatalog.lookup` returns `undefined` for them by design. Any UI reading
  `model.metadata` must degrade to the bare `{ id, name }` row rather than render an empty unit.
  Azure *could* be matched — `azure-provider.ts` discards the upstream `model` field, which is the
  underlying base model id.
- **Bedrock ids arrive region-prefixed.** `bedrock-provider.ts` returns an inference-profile id
  (`us.anthropic.claude-…-v1:0`) when one exists, but models.dev keys the foundation id
  (`anthropic.claude-…-v1:0`). The lookup strips `us.`/`eu.`/`apac.`/`global.` and keeps the `:N`
  version suffix, which is part of the upstream key.
- **`PROVIDER_MAX_CONTEXT_TOKENS` is still a per-provider guess and still drives compaction.** The
  catalog exposes the real per-model window on the API, but `aiProviderUtils.getMaxContextTokens`
  was not rewired: all five call sites (`shouldCompact` / `compactMessages` in `ee/agent/agent-compaction.ts`,
  `runawayTokenCeiling` / `boundContextForStep` in the worker's `run-agent-turn.ts`) thread
  `provider` and no `modelId`. So EE agent compaction still fires at, say, 200k for every Anthropic
  model including the 1M ones. Fixing it is plumbing plus an `agent-evals` run.
- **AI is billed on what the call cost, and the weight tables are gone.** `MANAGED_MODEL_WEIGHTS`,
  `DEFAULT_MANAGED_MODEL_WEIGHT`, `CHAT_BYOK_CREDIT_WEIGHT` and `ACTIVEPIECES_CHAT_TIERS[].creditWeight`
  were a static pricing ladder calibrated at about a thousand tokens per call, so a long call billed
  the same as a short one. Every language model is now built through `billedLanguageModel`
  (`packages/server/utils/src/activepieces-ai-cost.ts`): the managed provider bills the dollar cost
  OpenRouter reports divided by `AP_AI_CREDIT_USD_VALUE`, and any other provider bills one flat credit.
  Tool calls stay at one credit each. **The own-key credit is per model call for a direct AI step and
  per turn for an agent** — an agent turn is many model round-trips, so charging it per call would have
  silently raised the price of every agent on a customer's own key. The caller says which it wants with
  `ownKeyCredit`, and the agent's turn credit is charged in `chatToolBilling.chargeForLatestTurn`
  alongside its tool calls. **The worker does not know the edition, so the one own-key credit is
  reported on Community too; the API is what drops it.** `aiChargeFor` returns fixed credits for any
  non-`ACTIVEPIECES` provider, and only the CLOUD and ENTERPRISE arms of `app.ts` set the Autumn
  `billingProvider`; the default provider's `trackFeature` is a no-op, so Community records nothing
  and its `getCreditsAndAppSumoState` never blocks. Enterprise records the credit but
  `shouldBlockRunOnCredits` returns false for that edition, so only Cloud can refuse an AI call for
  being out of credits. Self-hosters on their own OpenRouter key pay OpenRouter directly, not us.
- **OpenRouter only reports `usage.cost` when the request asks for it.** The provider sends
  `usage: this.settings.usage` in the request body and copies `cost` into `providerMetadata` only when
  the response carries one, so without `usage: { include: true }` every managed call reports no cost,
  bills nothing, and raises no error. `openRouterModelSettings` in `packages/server/utils/src/ai-utils.ts`
  sets it for the ACTIVEPIECES provider and a test pins it. A call that still reports no cost bills
  zero, is counted, and pages on-call — there is deliberately no fallback to a table.
- **Ask AI's conversation history is hard-capped at 512 KB, on the write path.** The history lives in
  `context.store` under `ask-ai-conversation:<key>`, and the store-entry controller refuses a value over
  `STORE_VALUE_MAX_SIZE`. So the history cannot grow unbounded and cannot outgrow a request body — no
  separate limit is needed wherever it is passed along. An older version of the action could write a
  `{ messages: [] }` wrapper instead of a bare array, so any reader has to accept both shapes.
- **A failed `enrichWithKeysIfNeeded()` is self-sustaining, and it takes chat down with it.** `createKey` runs on the chat hot path — `chatHelpers.resolveChatProvider` → `getChatProvider` calls it whenever the platform's managed ACTIVEPIECES row has no `apiKey` — and the `save` happens *after* the OpenRouter call, so a failure persists nothing and the next chat turn calls `createKey` again. The mint runs under a per-platform `distributedLock` and re-reads the row inside it, and the write lands only if `auth` still matches the ciphertext it read (a `jsonb` compare; the row is then read back, since `affected` is unreliable on PGlite). A key that loses that write, or whose write throws, is revoked with `openRouterApi.deleteKey`. Two leaks stay open until a reconcile sweep over `listKeys` exists: a crash between `createKey` and the write, and a `createKey` that times out after OpenRouter has made the key. Seen in prod 2026-07-30: `keys-modify-api-rpd-v2` 429 (OpenRouter's key create/modify bucket, 10k/day on the provision key — a *separate* limit from inference), which killed every chat turn for the affected platform in `getChatConfig` before the first token, with no recovery until the bucket reset at 00:00 UTC.
- **A freshly minted managed OpenRouter key can answer 401 "User not found." for up to about 20s, so `enrichWithKeysIfNeeded` only saves a new key once it answers.** In prod that failed the first chat turns of about 15% of new platforms (AI-158). `GET /api/v1/key` is not a good enough check: in a 150-key run (2026-10-27) 7 keys passed it 3 times in a row and then still failed 1 to 3 completions. So `openRouterApi.isKeyLive` sends a 1-token completion and counts only a 401 or no answer as not live (so a retired or down probe model can't stall every new platform for 30s), and the wait needs 3 passes in a row 1s apart. With that, 1 of 150 keys failed one call afterwards, which chat's single retry absorbs (median wait 5.4s, p95 22s). How long a key takes varies with OpenRouter's load, and hammering it with 5+ new keys at once makes keys flap, so measure at most 3 at a time. The wait runs for up to 30s inside the mint lock, *before* the save, so a concurrent call still sees an empty row and queues on the lock instead of reading a key that isn't live (lock timeout 60s, so waiters outlast a 15s mint plus the wait plus one 10s check). On timeout it logs `waitUntilKeyIsLive` and saves the key anyway.
- **`openrouter-api.ts` has no retry, and every failure is a plain `Error`.** Calls go through `safeHttp.axios` with a 15 s timeout, and `openRouterRequest` rethrows any failure as `[OpenRouter] <METHOD> <path> error: <status> <body>`, so a caller cannot tell a 429 from a 500 without parsing the message.
- **Chat model tiers are Activepieces-only.** `ACTIVEPIECES_CHAT_TIERS` (`fast`/`smart`/`premium`, shown as Fast/Expert/Heavy) hold OpenRouter-shaped Anthropic ids, so they only mean anything for the ACTIVEPIECES and OPENROUTER chat providers. Any provider that declares `ALLOWED_CHAT_MODELS_BY_PROVIDER` (openai, anthropic, google) picks a real model id from that list instead. Naively stripping the tier's vendor prefix for every provider is what once sent `claude-haiku-4-5` to OpenAI and broke every message.
- **A short model list in the key's picker is the vendor's catalog, not a truncation.** `listModels` returns whatever the provider's own `/models` endpoint gives and filters nothing except the key's `modelScope` allow-list. Anthropic ships roughly a dozen models, OpenAI ~80 (mostly embeddings/tts/whisper), while OpenRouter is an aggregator and returns 400+ from every vendor it proxies — so the counts differ by an order of magnitude by design. Anthropic pages at 20 by default, which is why its request pins `?limit=1000`. If the list shows exactly three Claude models, that is the *chat* dropdown reading the curated `ANTHROPIC_CHAT_MODELS`, a different surface from the admin picker.
- **Read the chat model list through `aiProviderUtils.getCuratedChatModels({ provider })`.** It is the one accessor the server resolver (`agentHelpers.resolveModelIdForProvider`) and the chat dropdown share, so the two cannot drift; it returns `{ id, label }` or `undefined` — never an empty list, so callers may treat a result as non-empty. Labels come from the hardcoded `CHAT_MODEL_LABELS` (falling back to the id) rather than `AIProviderModel.name`, because the live `listModels` response cannot supply one for every provider: anthropic returns `display_name` and google `displayName`, but OpenAI's `/v1/models` returns ids only.
- **`conversation.modelName` carries either a tier id or a real model id** — it is a free string with no discriminator. A legacy tier id resolves to the tier's equivalent model when the provider ships it, else the provider's first curated model, so old conversations keep working after a provider switch. Note `premium` maps to opus 4.8, which the native anthropic list does not carry, so a legacy `premium` on anthropic lands on Sonnet. A new web chat sends no model at all until the user picks one (`use-chat.ts` starts `modelName` at null): the server resolves the published chat default for that turn and the worker stamps the tier it ran onto the conversation, so the chat default lives in one place, the published file, and the picker shows it as selected by reading `chat.defaultTierId` from the tiers endpoint. Before that change the web sent `smart` explicitly on every new chat, which is why a published `defaultTierId` never reached web chats. **That holds only where the picker shows tiers.** An own-key provider with a curated model list shows concrete models, but a null model still resolves the published chat default tier and then that tier's `nativeModelId`, which is in the Anthropic curated list: a published default of `fast` ran Haiku on every Anthropic-key chat while the picker showed Sonnet, and the worker's tier stamp kept it there. Since 7b the chat uses the shared `ModelPicker` and a new chat sends the pick the pill shows (`model-options` `defaultChoice` when untouched); agent chats send none, since the agent's job carries its own model. A message sent before the options load goes out with no pick and the server's `defaultPick` resolves it. Caught in review on #15838.
- Chat-provider resolution is **first `enabledForChat` row wins, else the visible managed ACTIVEPIECES row** — `pickChatRow`, shared by `findAvailableChatProviderRow` and both list endpoints. So a platform with `[openai, activepieces]` both chat-enabled still resolves to **openai** (an explicit pick beats the fallback), while a platform with nothing flagged resolves to the managed row instead of to nothing. The flag a list endpoint reports is therefore the *resolved* one, not the stored column — `ai_provider.enabledForChat` can say `false` on the row chat actually runs on, and `ensureManagedProviderRow` no longer writes it. The web no longer reads it (7b removed `useChatProvider`). There is no Default AI key control any more: it is derived from the platform's **Default chat model** setting (the "Default chat model" card on Providers), so it always matches the model new chats start on. A chat with no pick and no usable tier stores the Default AI key's first curated model (the pill's default); if that key's models can't be listed it keeps the legacy path rather than failing. A project entry is chat-enabled when it owns the picked row, whichever of its keys carries the flag: ranking (`selected` > `except` > `all`, newest first) and chat selection answer different questions, so reading `rows[0].enabledForChat` makes the client report "no provider configured" whenever the chat-enabled key is not the ranking winner. `findBy()` has no `ORDER BY`, so "first" is not guaranteed stable when several providers are chat-enabled.
- Listing providers is **not a pure read**: both `listConfigs` and `listForProject` go through `listVisibleRows`, which inserts the ACTIVEPIECES provider row when `aiCreditsEnabled && !activepiecesExists`. A `GET /v1/ai-providers` can therefore create a row. It also applies the hidden-provider filter (`plan.embeddingEnabled` hides the managed provider), which is why the client can trust its output without re-checking flags.
- **A managed chat turn is charged in two events, not one.** The model cost arrives per model call from
  the billing wrapper, and the turn's own-key credit and tool calls are charged by `chatToolBilling.chargeForLatestTurn`
  — which also owns the rule for which tool calls count (`mcp__*` plus a short allow-list, completed only).
  Both land on `CreditUsageSource.CHAT`, so the total is unchanged, but no single event holds a turn's
  whole cost. There is no fixed per-model price left to show a user: the model picker and the Credits FAQ
  now say the charge follows the call.
- Azure model listing (`azureProvider.listModels`) is pinned to the retired data-plane api-version `2023-03-15-preview` — newer versions 404 and break `validateConnection`; the configured `apiVersion` only affects inference via `@ai-sdk/azure` (GIT-1310).
- **`@activepieces/ai-providers` is `ai@7`, so only `ai@7` code may call it — a piece must build its own language models.** The shared `createLanguageModel` factory pins `ai@7` / `@ai-sdk/openai@4` / `@openrouter@3`, while `packages/pieces/community/ai` (and the engine that runs pieces) sit on `ai@6` / `@ai-sdk/openai@3` / `@openrouter@2`. #14446 pointed the piece at the factory anyway, which broke it two ways: `tsc` rejects the result (`specificationVersion "v4" is not assignable to "v2"`, `LanguageModelV4` vs the piece's `LanguageModelV2 | V3`), and at runtime the piece's own `ai@6` `generateText` would refuse a v4 model. CI hid it because the pieces build only runs when a PR's diff touches `packages/pieces/**`, and a post-merge run on main diffs `HEAD` against `origin/main` — empty, so no piece is ever built there. Fixed by giving the piece back a local `buildLanguageModel` switch built on its own SDKs; the factory is server-side only until the pieces move to AI SDK 7. Cloudflare Gateway is the one provider the factory refuses (`throw`) because its routing is caller-specific; each caller builds that one itself. **The standing consequence: a new provider must be added to BOTH switches** — `createLanguageModel` in `packages/core/ai-providers` *and* `buildLanguageModel` in `packages/pieces/community/ai/src/lib/common/ai-sdk.ts` — and the piece one is the easy half to forget, because nothing before step execution touches it. Miss it and the provider connects, validates, lists its models and saves without complaint, then every AI-piece step dies at run time on the switch's `default:` with `Provider <name> is not supported`. Since pieces cannot import `@activepieces/shared`, anything the piece-side case needs (base-url maps, config types) also has to be re-exported through `packages/pieces/framework/src/index.ts`. The `git show --stat` of the commits that added Mistral (#13088) and Bedrock (#12712) is the reliable checklist for what a provider actually touches — both include `ai-sdk.ts`.
- **Attribution headers are for `ACTIVEPIECES` only, and go through the factory's `extraHeaders` option rather than a local `createOpenRouter` call.** The managed provider is OpenRouter under the hood on our own key, so the `x-ap-*` headers are what tag *our* account's events: `x-ap-platform-id` / `x-ap-conversation-id` / `x-ap-run-id` on the agent path, `x-ap-project-id` / `x-ap-flow-id` / `x-ap-run-id` on the piece path. BYOK `OPENROUTER` is a customer's own account and must not get them. Constructing the provider inline to attach headers is also what silently drops `openRouterSettings` (the web-search plugin), since the factory is the only place that still passes them. (`CUSTOM` separately receives the piece-path metadata headers — that is #11700's metadata forwarding for self-hosted OpenAI-compatible endpoints, older than either the rename or the Autumn work and unrelated to OpenRouter attribution. Its precedence is deliberate: admin-configured `defaultHeaders` override the `x-ap-*` metadata, and the api key is applied last.)
- **`mistralViaOpenRouter` does not mean "the managed provider"; it is read only inside the `MISTRAL` case, and that branch looks like dead legacy.** `ACTIVEPIECES` routes through OpenRouter unconditionally and ignores the flag, so the only thing the agent path's `mistralViaOpenRouter: true` does is send a `MISTRAL` chat row to openrouter.ai — carrying that row's *Mistral* key, which cannot authenticate there. `MISTRAL` also has no `ALLOWED_CHAT_MODELS_BY_PROVIDER` entry, so `getCuratedChatModels` returns `undefined` for it and the resolver falls back to a tier's OpenRouter-shaped id. The fall-through arrived as a drive-by in #13489, not as a routing decision. Don't infer "this provider is AP-managed" from that case group.
- **AI Tool Configs** are a *sibling* feature (same `ai/` dir), distinct from AI Providers: they give the chat assistant external capabilities via `/v1/ai-tools` (platform-admin, EE/Cloud). **AiToolCapability** = `WEB_SEARCH`/`WEB_SCRAPING`/`IMAGE_GENERATION`; **AiToolProvider** = `TAVILY`/`FIRECRAWL`/`APIFY`/`FAL`. One config per capability (unique on platformId+capability); consumed by chat via `getEnabledTools()`. **Because the config is per-platform, it can never serve a first-run flow on Cloud.** A self-serve signup lands on a brand-new platform with no configs at all, so `getEnabledTools()` returns `{}` for exactly the users a new-signup feature is aimed at, and any capability read from it silently no-ops rather than failing loudly. A capability that has to work for someone who just signed up needs a cloud-wide `AppSystemProp` key instead, the way `TURNSTILE_SECRET_KEY`, `FEATUREBASE_API_KEY` and `APPSUMO_TOKEN` are sourced. Note there is no `ENRICHMENT` capability here, so anything needing people or company enrichment has nowhere to read a key from today.
- **`/v1/ai-tools` is registered only in the CLOUD and ENTERPRISE branches of `app.ts`, but the AI Center page that reads it is not edition-gated** — so a Community admin opening the Capabilities tab fired `useAiToolConfigs`, got Fastify's `Route not found`, and the page showed the global "Failed to load data" dialog of the day. Shipped that way from #13911 until the tab was gated on `ApFlagId.EDITION` in the page. What made this class of bug hard to place is that the surface was then page-independent (raised from `QueryCache.onError` in `query-client.ts`) and React Query's 3 default retries meant it landed several seconds later on whatever page you navigated to next (the report was against `/platform/general`). The tab now renders `DataFetchErrorState` in place instead, so a repeat would at least accuse the right page. When a screenshot's edition is in doubt, read the sidebar: **Billing & subscription** and **Usage** carry a lock only when `edition === COMMUNITY`, every other lock there is plan-driven. Any new EE-only route needs its UI entry point gated the same way, `enabled:` on the query or hiding the surface.
- **The AI Center providers page renders its whole state machine off one query, so every branch needs an `isLoading` gate above it.** `ProvidersTab` derives the empty state, the grouped list *and* the `?config=<id>` detail view from `useAiProviderConfigs()`. While that query is pending, `data` is `undefined` — `configs` is `[]`, so the empty "Connect your first provider" screen paints for a frame, and a hard refresh on a `?config=` URL finds no matching row and falls through to the list before flipping to the detail page. The parent `SuspenseWrapper` does not cover it: the route chunk and the suspense-backed projects collection resolve first, then the component mounts with no providers yet. Any new branch added to this page belongs below the `if (isLoading) return <ProvidersSkeleton />` gate, not beside it.
- **A save bar in the AI Center needs a full-height column, not `fixed` and not bare `sticky`.** The page sits inside *two* nested scrollers — `PlatformLayout`'s `flex flex-col flex-1 overflow-auto` and the AI Center's own `flex-1 overflow-auto` column — so `fixed inset-x-0` centres the bar on the viewport, roughly a sidebar-width left of the content column it belongs to. Swapping it to `sticky bottom-4` fixes the centring for free but pins nothing when the content is shorter than the viewport: the bar just lands wherever the content ends, mid-page. The working shape is `min-h-full` + `flex flex-col` on the tab-content wrapper, `flex-1` on the `TabsContent`, `grow` on the panel root, then `sticky bottom-4 mt-auto` on the bar — `mt-auto` handles the short page, `sticky` handles the long one. `min-h-full` resolves only because every ancestor up to `SidebarInset h-full` has a definite height; that chain is what makes the whole thing work, so check it before reusing this elsewhere.
- **A chat *tier* is our own idea, so it has no meaning on a provider we ship no curated model list for — resolve it by asking the key what it offers.** `ACTIVEPIECES_CHAT_TIERS` (fast/smart/premium) only maps to a real id through `ALLOWED_CHAT_MODELS_BY_PROVIDER`, which covers Anthropic, OpenAI, Google, Vertex and Activepieces. Everything else — Bedrock, Azure, Mistral, xAI, DeepSeek and the other OpenAI-compatible vendors — has an empty candidate list. The resolver used to invent an id from the tier (strip the vendor prefix, dots to dashes: `anthropic/claude-haiku-4.5` -> `claude-haiku-4-5`), which Bedrock rejected as "The provided model identifier is invalid" (GIT-1840, Pylon 5836). #15355 removed the invention and left a throw in its place, which was worse: every agent turn on those providers died with `ENTITY_NOT_FOUND: this AI provider key allows no text model a chat turn can run on` before the configured model handled anything (Pylon 6069). **The missing ingredient was never a smarter guess, it was the key's own model list.** `agentHelpers.resolveModelId` is now the single resolver for every surface: it tries our catalogue first and, only when that has nothing to say, reads `aiProviderService.listModels` (process-cached, cleared nightly by cron, already filtered by the key's `modelScope`), preferring a model whose id *contains* the tier's `nativeModelId` — Bedrock spells them `eu.anthropic.claude-sonnet-4-6-...`, so a tier still selects the model it names wherever the vendor hosts it, and otherwise degrades to the first text model the way `pickAllowedModel` already does for an admin-listed catalogue. Picking from a list the provider returned is the opposite of the fabrication #15355 removed. **Two traps, both learned the hard way.** First, an agent surface and generic chat are not symmetric: an agent *carries* a real model id (its picker lists `listModels` output unfiltered, precisely because these providers have no `ALLOWED_CHAT_MODELS_BY_PROVIDER` entry) while chat carries only a tier name, because the pre-7b chat selector fell back to tier options when `getCuratedChatModels` returned `undefined`. Second, **being listed proves the key can see a model, not that the account can invoke it** — so where a model is already settled the fast round reuses it rather than choosing again (`getAgentConfig` passes the model the turn resolved; the tool RPC gets the run's own model from the worker, `modelId` on `executePieceTool`). One more trap, found in review on 2026-09-27: a stored name is not the model the run ran on. The worker stamps the resolved *tier id* on an agentless inline run's conversation, so reading it back as a model hands a bare `smart` to a catalogue-less own-key provider, and no rule on the string can save it, because a deployment may be named `smart` too. So nothing is read off the conversation or the agent for the fast round any more: the worker says which model the turn resolved, `modelId` on `ExecutePieceToolRequest` (core-execution), and the API borrows exactly that. The field is required, not optional, because the dispatch gate keeps worker and app on one release, so an older worker never reaches a newer API. Resolving the fast tier independently once picked a Haiku the account could not invoke and failed a turn whose configured Sonnet worked fine. A borrowed model still has to pass the key's current `modelScope`, or resolution falls through to the list, which applies it. Nothing crosses the worker, so a rolling deploy has no version-skew window.
- **Non-chat models are filtered out centrally in `fetchModels`, not per provider strategy.** OpenAI's `/v1/models` returns `whisper-1`, `tts-1`, `gpt-4o-transcribe`, `text-embedding-3-*`, `omni-moderation-latest` and `sora-*`, and every strategy that cannot tell them apart typed them `TEXT` — so they were offered in Ask AI / Summarize / Classify / Extract and died at run time on chat completion (#14496). One `aiProviderUtils.isChatModelId` denylist now runs inside `fetchModels`, which is the single choke point behind both the project route and the admin `configs/:id/models` route, so a new provider is covered without opting in. Three things about it are load-bearing: it tests **only `TEXT`-typed rows**, so image models are untouched; it **never judges a name the customer chose** — admin-typed configs (`'models' in config` — VERTEX, CUSTOM, CLOUDFLARE_GATEWAY) skip it because an admin who typed an id meant it, Azure skips it (`modelIdsAreCustomerNamed` on the strategy) because it lists deployment names like `support-voice-agent` and filters on `deployment.model` itself, and a fine-tune is judged by its base model only (`ft:<base>:<org>:<label>:…`, Azure `<base>.ft-…`) so an org called `voiceflow` or a `content-moderation` label does not hide a working chat model; and `validateConnection` **bypasses `fetchModels` entirely**, so a key that lists nothing but whisper still validates as a working key. The rule is a denylist on purpose — an unknown id is treated as chat, so a self-hoster's own model on a custom endpoint never disappears, at the cost of a new modality leaking until a pattern is added. Because pieces may not import `@activepieces/core-*` (eslint `no-restricted-imports`), the openai piece's `isLLM` reaches the same ruleset through a `packages/pieces/framework/src/index.ts` re-export. Still leaking, knowingly: vendor-branded generative models on the five compat vendors that publish no modality data (`cogview-*`, `wan*`, `music-*`, `video-01`), whose real ids nobody could verify without five paid keys — xAI is handled structurally, see the next bullet.
- **xAI is the one compat vendor asked what its models output, because it is the one that answers.** `listVendorModels` hits `/v1/language-models` for XAI first and keeps a model only when `output_modalities` includes `text`; the OpenAI-shaped `/v1/models` that the other five vendors use carries no modality field at all, which is why `grok-2-image-1212` and the Grok Imagine image/video models were offered as chat. A model that declares no modalities falls back to the id denylist. Every failure mode degrades to the old path rather than emptying the picker: a 404, a changed shape, or an empty `models` array all fall through to `/v1/models`, and because that fallback is what authenticates, a bad key still fails validation with the vendor-prefixed error. The endpoint pair (`/language-models`, `/image-generation-models`) was already known in-repo — `packages/pieces/community/grok-xai` calls both. Adding a vendor to `VENDORS_DECLARING_MODALITIES` is the whole opt-in; do it only with a real `/models` response in hand, not from recollection of a catalogue.
- **Google is filtered on `supportedGenerationMethods`, and that is what keeps Imagen out of the image picker.** A model survives only if the field is absent or includes `generateContent`, which is the only way to exclude `aqa` (`generateAnswer`) and `gemini-embedding-001` (`embedContent`) — no name pattern catches them. The side effect is deliberate: `imagen-*` was classified `IMAGE` (because `'imagen'.includes('image')`) and offered in Generate Image, but the AI piece reaches Google image generation through `generateText`/`:generateContent` (there is no `google` case in `buildNativeImageModel`), and Imagen only answers `:predict` — so those options always failed at run time. Gemini image models do expose `generateContent` and are unaffected; Vertex is where Imagen actually works, and it is admin-typed so the filter never sees it.
- `packages/server/api/src/app/ai/` — backend module: provider controller, service, entity, module, plus the sibling ai-tool-config files
- `packages/server/api/src/app/ai/providers/` — per-vendor strategies keyed by `AIProviderName`
- `packages/server/api/src/app/platform/billing-provider.ts` — `assertCreditsAndAppSumoNotExceeded` credit gate
- `packages/server/api/src/app/ee/platform/platform-plan/openrouter/openrouter-api.ts` — the OpenRouter provisioning client (`createKey`/`updateKey`/`getKey`/`listKeys`/`deleteKey`)
- `packages/core/ai-providers/src/lib/create-language-model.ts` — the shared per-provider model factory (`createLanguageModel`, `buildOpenAICompatibleHeaders`) used by both the AI piece and the agent path
- `packages/core/shared/src/lib/management/ai-providers/index.ts` — shared zod schemas, enums, request/response types
- `packages/core/shared/src/lib/management/ai-tools/index.ts` — shared schemas for the AI Tool Configs sibling
- `packages/web/src/features/platform-admin/api/` + `packages/web/src/features/platform-admin/hooks/` — frontend API clients and TanStack Query hooks (`ai-provider-*`, `ai-tool-config-*`)
- `packages/web/src/app/routes/platform/setup/ai/` — the AI Center: `providers-tab/` (provider groups, connect dialog, per-config detail panel with model and project scope pickers) and `capabilities-tab/`
- `packages/web/src/app/routes/platform/setup/ai-capabilities/` — admin page, capability dialog, provider catalog for AI Tool Configs
- `packages/web/src/features/agents/ai-model/` — model selector used in agent step settings

Paths verified 2026-07-26.
- **`OPENROUTER_PROVISION_KEY` is a management key and cannot make inference calls; only the per-platform keys it mints can.** OpenRouter states it outright: management keys are for creating, listing, updating and deleting other keys and are rejected by every completion or decisions endpoint. So "we already have an OpenRouter key" is true for provisioning and false for anything that needs to call a model at the instance level (the AI Router's Jev call, for example): that needs an ordinary inference key from the same account, created in the dashboard or via the same `POST /keys` the instance already uses, with a credit `limit`. The AI Router does exactly that since 2026-09-23: it runs on the platform's managed key where credits are on, else the admin's own OpenRouter row, as a worker `EXECUTE_AI` job billed like any AI step.
- **`z.enum(SomeTsEnum).exclude([SomeTsEnum.MEMBER])` compiles and excludes nothing at the type level; pass the key string, `.exclude(['MEMBER'])`.** zod's `exclude` is typed over `keyof T` and returns `ZodEnum<Omit<T, U[number]>>`. A TS enum member is its own literal type, not the string `'MEMBER'`, so `Omit` finds no key to drop and the inferred value type keeps every member, while the runtime check does exclude it. The symptom is one step removed from the cause: a switch over the parsed field is suddenly non-exhaustive (`switch-exhaustiveness-check` in lint, `TS2366` in the build) even though the branch you removed can never arrive. `/v1/ai/execute` hit this when `ROUTE` was carved out of `AiStepAction` for #15707. Neither vitest nor a package-local eslint run without type information reports it; only the build does.
- **The managed Activepieces row can never restrict its models, so a finding that the managed picker offers a model the key forbids is not reachable.** The row is inserted with `modelScope: 'all', modelIds: []`, `create()` refuses `provider === ACTIVEPIECES` (`aiProvider.activepiecesIsManaged`), and `update()` handles only `enabledForChat` for it and returns before any scope field is read (`ai-provider-service.ts`). Every published tier therefore runs on it, and any `modelScope` filtering on the managed path is inert. Reviewers (Greptile on #15871) raise this as a P1; answer with these three facts rather than adding a filter.
- **OpenAI, Azure and Meta language models run on the Responses API with `store: false`; everything else OpenAI-shaped stays on Chat Completions.** New OpenAI-shaped vendors default to Responses whenever their API serves an OpenAI-compatible `/responses` (Meta's does; NVIDIA's hosted API serves only `/chat/completions`, so NVIDIA stays on chat). `createLanguageModel` builds OpenAI and Azure with `.responses()`. A Cloudflare Gateway `openai/*` text model goes to the gateway's `/openai` endpoint with `.responses()` on **both** routes: chat and agents use `routing: 'compat'`, which otherwise sends everything to `/compat/chat/completions`, so the OpenAI check has to sit before the compat early return. `withoutServerStorage` wraps them so `store: false` is the default. Chat Completions refuses tools + reasoning from gpt-5.4 on, and the gpt-5.6 family needs Responses for agents and chat (GIT-1848). `store: false` matters because the default `true` makes the SDK replay saved history as `item_reference` ids, which break after OpenAI's 30-day retention, on a key switch, or when compaction drops parts. Azure's Responses model reads `providerOptions.azure` and only falls back to `openai` when that key is absent, so set options under the provider's own key. The vendors (xAI, DeepSeek, ...), Mistral, every non-OpenAI model on the Cloudflare compat route, and Custom keep Chat Completions, because many of them only speak it. Custom offers Responses through `apiStyle: 'responses'`. To test `store: false`, give the saved assistant turn a **text** part with an `itemId`. With `store: true` the SDK swaps only text, reasoning and provider-run tool parts for `item_reference`, and plain function tool calls are always sent in full, so a tool-call-only fixture passes either way.
- **No provider ever receives a forced `tool_choice`.** Meta's Model API accepts only `auto`, and Claude Opus 5.5, Sonnet 5.5, Fable 5.1 and Mythos 5.1 answer `any`/`tool` with a 400 on Anthropic, Bedrock and OpenRouter alike (AI-174), so forcing is a contract we enforce, not a provider setting. Two callers force a tool choice: Extract structured data (`required`) and the agent's last step (a named tool, or `none`). `aiUtils.createModel` wraps every language model in `withAutoToolChoice` (`autoToolChoiceOnly`), which sends `required` as `auto`, a named tool as `auto` with only that tool offered, and `none` as no tools. `auto` alone lets the model answer in text, so a forced call also gets an instruction naming the tool and one retry when the reply has no tool call, and a second text reply throws `ForcedToolNotCalledError` so callers fail instead of completing without the tool output; a forced streamed call (the agent's last step) is served from that retried generate call through `simulateStreamingMiddleware`. `withAutoToolChoice` wraps the *billed* model, so billing sees each attempt as its own call with its own `providerMetadata` cost. Wrapped the other way round, managed billing reads the dollar cost off the returned result only, and the first attempt goes unbilled. Any middleware that retries belongs outside `billedLanguageModel` for the same reason. Do not `import` the error class into `server-utils`: CI installs two copies of `ai@7.0.44` with different peer hashes, and pulling `ai-providers`' types in first makes the global `AI_SDK_TELEMETRY_INTEGRATIONS` come from the other copy, which fails the `server-utils` build in `agent-ai-utils.ts` (it passes locally, where bun dedupes them). The instruction goes in as a `user` message: Bedrock and Google reject a `system` message after the conversation has started. Build models through `aiUtils.createModel`, never the raw factories, or the rule is skipped. The Llama API (`api.llama.com`) that predates it shut down on 2026-07-06; Meta now means the Muse models on `api.meta.ai/v1`.
- **Cohere runs on its native v2 chat API (`@ai-sdk/cohere`), not its OpenAI compatibility API.** The compatibility API (`api.cohere.ai/compatibility/v1`) takes no `tool_choice` and has no `/models`, so Extract structured data and the agent's last step would break there. The native API accepts a forced tool as `REQUIRED`, though callers never send one (see `withAutoToolChoice` above). Models are listed from `GET api.cohere.com/v1/models?endpoint=chat`, minus deprecated ones. There are no Cohere embeddings, because the knowledge-base column is 768-dim and Cohere's start at 1024. Most Cohere models cap output at 4k–8k tokens, while chat asks for 32k unless the catalog knows the model. `UNKNOWN_MODEL_OUTPUT_CEILING` in `agent-ai-utils.ts` caps an unknown Cohere model at 4k. On long multi-tool turns Cohere can call tools that don't exist and end the stream with `finish_reason: ERROR`, and `@ai-sdk/cohere` 4.0.16 fails to parse that event ("Type validation failed").
- **NVIDIA's hosted keys (build.nvidia.com) are for trial and development only, and its `/models` lists classifiers next to chat models.** NVIDIA's terms forbid production use of the hosted catalogue (about 40 RPM, no paid tier), so production users run NIM themselves and connect it through Other (OpenAI Compatible). The catalogue's ids carry no capability field, so the `guard`, `safety`, `reward`, `parse`, `clip`, `detector`, `calibration` and `deplot` fragments in `NON_CHAT_MODEL_ID_FRAGMENTS` are what keep guard, reward and parser models out of chat pickers.
- **Capability cards draw a service's logo from `AI_TOOL_CATALOG` (`setup/ai-capabilities/catalog.ts`, `logoUrl`), not from a piece.** Tavily, Firecrawl and Apify reuse their piece logos on the CDN; fal.ai has no piece, so `pieces/fal-ai.png` 404s and the card shows an "F" until someone uploads it. A new capability also needs a group in `capabilityGroups()` and a colour in `CAPABILITY_LOOK` (`setup/ai/capabilities-tab`), or it never renders.
- **Replacing a key's credentials remounts the key page.** `ConfigDetail` is keyed `${id}:${credentialsVersion}` and `onConnected` bumps the version, so any unsaved edit on the page would vanish while the toast says "Saved". The Replace button is therefore disabled while the page is dirty, and `?tab=settings` (read into `initialTab`) brings the admin back to Settings after the remount. It lives in the URL, not state, so leaving by the breadcrumb or Back and reopening the key starts on Models again. Anything else that refreshes the key from outside the page needs the same guard.
- **A saved key whose provider the running build doesn't know crashes the Capabilities page.** `aiCapabilitySources` (`setup/ai-capabilities/catalog.ts`) reads `AI_PROVIDER_CAPABILITIES[provider]` with no guard, so a row left by a provider that exists only on another branch (or was removed) throws `Cannot read properties of undefined (reading 'supportsEmbedding')`. Locally it shows up after testing a provider branch and switching back: delete that key or return to the branch. Removing a provider for real needs a migration or a guard first.
