Operations & reliability

JARAI’s pipeline is asynchronous and self-healing: transient failures retry, failing providers are routed around, and permanent failures are surfaced as alerts and parked on a dead-letter queue for an operator. This page covers the tools that keep productions flowing and tell you when something needs a human.

🎬 Operations & reliability tour

Prefer to read? Open the step-by-step transcript

1. Dashboards → Alert operations — open alerts by type and severity.
2. Settings → DLQ Management — messages that failed permanently; inspect and resubmit.
3. Stuck-production detection — the reconciliation engine re-drives missing work it can rebuild; the orphan sweepers raise alerts (or fail a brief that can’t move) for the rest.
4. Provider health — failing providers are routed around so productions keep moving.

Retry vs dead-letter (how failures are classified)

Every pipeline step classifies its failures:

• Retriable (transient — SQL pool exhaustion, provider 5xx, lock lost, blob timeout): the message returns to the queue and the step status becomes Retrying. No action needed.
• Non-retriable (permanent — contract violation, missing required record, retries exhausted): the message dead-letters immediately and the step status becomes Failed, carrying a structured error envelope (error class/code, function, brief/step, correlation id).

Tip

This is why a single provider outage rarely breaks productions — 5xx is retriable and the model-selection chain also routes around a Failing provider.

Alerts & escalation

• Alert operations dashboard (Dashboards → Alert operations) shows open alerts grouped by type and severity, sourced from the AlertQueue.
• The AlertDispatcher routes alerts to their channels; the EscalationFunction raises severity for alerts left unacknowledged past their SLA; a periodic digest summarises activity.
• Acknowledge an alert once you’ve actioned it so escalation stops.

Dead-letter queue (DLQ)

Settings → DLQ management lists permanently-failed messages. For each you can inspect the error envelope and resubmit once the root cause is fixed.

1. Read the error — the envelope tells you the error class/code, function, and the production/step it belongs to.
2. Fix the cause — e.g. a missing credential, a bad template/contract, or a provider misconfiguration.
3. Resubmit — the message re-enters its topic and the step re-runs from a clean state (idempotency guards prevent double-processing of already-complete steps).

Caution

Resubmitting before fixing the cause just dead-letters the message again. Confirm the fix first.

Stuck-production recovery

Three background jobs watch for productions that have stalled mid-pipeline (e.g. a message lost to an infrastructure blip):

• ReconciliationEngineFunction (every minute) compares each in-flight production’s expected and actual work against its reconciliation invariants and either re-drives the missing work or escalates.
• StepOrphanSweeperFunction (every 15 minutes) finds a step still Queued long after its predecessor completed — a lost advance message — and raises a StepOrphaned alert with the step’s blocking reason. It does not replay the message itself; that’s an operator action.
• BriefOrphanSweeperFunction (every 30 minutes) fails a production that is InProgress with nothing left running, so it stops looking alive.

Combined with idempotency guards on every step, many stalls self-resolve; the rest surface as alerts rather than sitting silently.

Provider health & quality

• ProviderHealthCheckTimer maintains the Healthy/Degraded/Failing/Suspended badge used by the model-selection chain (see AI providers & models).
• RapidQualityCheckFunction compares the mean quality score of an account’s last five productions with its 30-day baseline, and raises a QualityRegressionWarning when the drop exceeds the account’s regression threshold — or when the recent mean falls below the account’s optional absolute quality floor. It’s advisory: it alerts, it doesn’t pause anything.

Where to look when something’s wrong

Symptom	Look here
A production is “Failed”	Production detail → the failed step’s error; resubmit via DLQ after fixing
Many failures from one vendor	Provider health badge; the chain should already be routing around it
Productions not starting	Budget/concurrency gates (account limits) and provider health
Quality regressions	RapidQualityCheck alerts on the Alert operations dashboard