Self-hosting

29 min read·Updated 30 Jul 2026

Arble is designed to run entirely inside your environment. No component requires a managed cloud, and no telemetry leaves the perimeter unless you configure it to.

You choose where compute, storage, models and memory live. The same binaries run on a laptop and on a multi-node cluster — the difference is configuration, not a separate product edition.

Architecture

Arble separates into four planes: a stateless request plane, a stateful orchestration plane, an execution plane that needs isolation, and the data plane. Understanding which is which tells you what to scale and what to protect.

API Gateway terminates TLS, authenticates every request, and enforces rate limits before anything reaches the runtime. It is stateless and the only component that needs ingress. Run at least two replicas.

Arble Runtime owns sessions and runs. It accepts work, tracks run state, and coordinates the planner and workers. State lives in Postgres, so the runtime itself is horizontally scalable.

Planner decides which tool to call for a given step. It reads the tool registry and the session context, calls a model, and emits a tool call. It is CPU-light and latency-bound on the model, not on itself.

Memory stores durable per-project facts and serves semantic search. It owns embedding generation and the vector index. This is the component that benefits most from a GPU if you embed locally.

Tool Registry is the single index of every callable capability, regardless of origin. The planner sees built-in tools, Tool SDK tools, MCP tools and connector tools identically — one schema shape, one permission model.

Built-in Tools — filesystem, terminal, HTTP, browser — execute inside a sandboxed worker, never in the runtime process. A tool that escapes its sandbox must not land in a process holding database credentials.

MCP Servers run as separate processes or Pods. Remote servers are reached over HTTPS; local stdio servers are spawned per-worker, which is why stdio requires a self-hosted deployment.

LLMs are external to Arble. The runtime speaks an OpenAI-compatible protocol to whatever you point it at — local vLLM, Ollama, or a hosted provider. Arble does not embed a model.

Storage is Postgres for control-plane state, Redis for queues and coordination, and an S3-compatible object store for files. Each is a standard component you likely already operate.

Component summary

Deployment options

Pick the smallest topology that meets your availability requirement. Every option below runs the same images.

Desktop and workstation run a single daemon with SQLite and no gateway. There is no cluster to operate — this is what the desktop app installs.

Docker Compose is the right default for a team evaluation. One host, real Postgres and Redis, and a config file you can lift into Kubernetes later without rewriting.

Kubernetes is the right default for production. You get rolling updates, health-checked replicas and autoscaling without building them yourself.

Bare metal matters when you need hardware-level isolation for tool sandboxes, or when GPU passthrough through a hypervisor is unacceptable.

Avoid single-node Kubernetes as a production target. It adds operational surface without adding availability — Docker Compose is simpler and fails in more predictable ways.

System requirements

Sizing below excludes model inference. Add GPU capacity separately per Models.

Operating systems. Ubuntu 22.04 LTS and 24.04, Debian 12, RHEL 9 and derivatives are tested. macOS 14+ is supported for desktop and workstation only. Windows is supported through WSL2 for development, not production.

Kernel requirements. Tool sandboxing uses cgroups v2, user namespaces and seccomp. A kernel older than 5.15, or one with user namespaces disabled, falls back to weaker isolation and logs a warning at startup.

Storage characteristics. Postgres is latency-sensitive on writes; put it on NVMe or provisioned IOPS. Object storage is throughput-sensitive, not latency-sensitive — network storage is fine.

Plan roughly 2 GB of Postgres per 10,000 runs retained, and size the object store for files and screenshots, which dominate volume.

Docker

Single-host Docker is the fastest way to a working deployment. Use it for evaluation, or for production where one host is an acceptable failure domain.

Installation

Images are published to GitHub Container Registry, tagged by exact version. Pin the patch version in production — latest is for local experiments only.

docker pull ghcr.io/arble-ai/runtime:1.8.2
docker pull ghcr.io/arble-ai/gateway:1.8.2

Volumes and ports

Three paths must persist. Everything else in the container is disposable.

Configuration

Configuration is a YAML file, overridable by environment variables. Environment wins, which is what makes one image work across environments.

# /var/lib/arble/config/arble.yaml
server:
  bind: 0.0.0.0:8443
  external_url: https://arble.internal.example.com

database:
  url: postgres://arble@postgres:5432/arble
  max_connections: 40

redis:
  url: redis://redis:6379/0

storage:
  backend: s3
  endpoint: https://minio.internal.example.com
  bucket: arble-files

models:
  default: local-qwen
  providers:
    - name: local-qwen
      kind: openai_compatible
      base_url: http://vllm:8000/v1

telemetry:
  metrics: true
  otlp_endpoint: http://otel-collector:4317

Docker Compose

This stack is production-shaped on one host: real Postgres, real Redis, separated worker, no host networking.

services:
  gateway:
    image: ghcr.io/arble-ai/gateway:1.8.2
    ports: ["8443:8443"]
    environment:
      ARBLE_RUNTIME_URL: http://runtime:8080
      ARBLE_TLS_CERT: /etc/arble/tls/tls.crt
      ARBLE_TLS_KEY: /etc/arble/tls/tls.key
    volumes:
      - ./tls:/etc/arble/tls:ro
    depends_on: [runtime]
    restart: unless-stopped

  runtime:
    image: ghcr.io/arble-ai/runtime:1.8.2
    environment:
      ARBLE_DATABASE_URL: postgres://arble:${PG_PASSWORD}@postgres:5432/arble
      ARBLE_REDIS_URL: redis://redis:6379/0
      ARBLE_WORKER_URL: grpc://worker:7070
    volumes:
      - arble-config:/var/lib/arble/config
    depends_on:
      postgres: {condition: service_healthy}
      redis: {condition: service_started}
    restart: unless-stopped

  worker:
    image: ghcr.io/arble-ai/worker:1.8.2
    # Sandboxing needs these; it does NOT need privileged.
    security_opt: ["seccomp=./seccomp-arble.json"]
    cap_drop: ["ALL"]
    cap_add: ["SYS_ADMIN"]
    tmpfs: ["/tmp:size=2g"]
    restart: unless-stopped

  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: arble
      POSTGRES_PASSWORD: ${PG_PASSWORD}
      POSTGRES_DB: arble
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U arble"]
      interval: 10s
      retries: 5
    restart: unless-stopped

  redis:
    image: redis:7.2
    command: ["redis-server", "--appendonly", "yes"]
    volumes:
      - redisdata:/data
    restart: unless-stopped

volumes:
  arble-config:
  pgdata:
  redisdata:

Run migrations before starting the stack for the first time, and before every upgrade:

docker compose run --rm runtime arble server migrate
docker compose up -d

Networking and MCP containers

Compose puts every service on one bridge network, so only the gateway needs a published port. Local MCP servers join the same network and are reachable by service name. Register them against the internal hostname, not localhost:

  mcp-postgres:
    image: ghcr.io/arble-ai/mcp-postgres:1.2.0
    environment:
      DATABASE_URL: postgres://readonly@warehouse:5432/analytics
    restart: unless-stopped

docker compose exec runtime \
  arble mcp add postgres --url http://mcp-postgres:8080/mcp

Kubernetes

Kubernetes is the recommended production target. The Helm chart is the supported installation path; raw manifests are shown to explain what the chart produces.

Namespace and installation

kubectl create namespace arble
helm install arble oci://ghcr.io/arble-ai/charts/arble \
  --version 1.8.2 \
  --namespace arble \
  --values values.yaml

# values.yaml
gateway:
  replicaCount: 3
  ingress:
    enabled: true
    className: nginx
    host: arble.internal.example.com
    tls:
      secretName: arble-tls

runtime:
  replicaCount: 3
  resources:
    requests: {cpu: "2", memory: 4Gi}
    limits: {cpu: "4", memory: 8Gi}

worker:
  replicaCount: 6
  resources:
    requests: {cpu: "1", memory: 2Gi}
    limits: {cpu: "2", memory: 4Gi}

memory:
  persistence:
    size: 100Gi
    storageClass: fast-nvme

postgres:
  # Point at managed Postgres rather than running it in-cluster.
  external: true
  existingSecret: arble-postgres

models:
  default: local-qwen
  providers:
    - name: local-qwen
      kind: openai_compatible
      base_url: http://vllm.models.svc.cluster.local:8000/v1

Deployment

The runtime is stateless, which is what makes rolling updates safe. Note the probes — they are what make a rollout non-disruptive.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: arble-runtime
  namespace: arble
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels: {app: arble-runtime}
  template:
    metadata:
      labels: {app: arble-runtime}
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        fsGroup: 10001
      containers:
        - name: runtime
          image: ghcr.io/arble-ai/runtime:1.8.2
          ports:
            - {containerPort: 8080, name: http}
            - {containerPort: 9090, name: metrics}
          env:
            - name: ARBLE_DATABASE_URL
              valueFrom:
                secretKeyRef: {name: arble-postgres, key: url}
            - name: ARBLE_REDIS_URL
              valueFrom:
                secretKeyRef: {name: arble-redis, key: url}
          livenessProbe:
            httpGet: {path: /healthz, port: 8080}
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet: {path: /readyz, port: 8080}
            periodSeconds: 5
          # Runs can be long. Give in-flight work time to drain.
          lifecycle:
            preStop:
              exec:
                command: ["arble", "server", "drain", "--timeout", "120s"]
          resources:
            requests: {cpu: "2", memory: 4Gi}
            limits: {cpu: "4", memory: 8Gi}
      terminationGracePeriodSeconds: 150

/healthz reports process liveness only. /readyz additionally checks Postgres, Redis and the default model backend — a runtime that can’t reach its database reports unready and stops receiving traffic instead of failing requests.

Service and Ingress

apiVersion: v1
kind: Service
metadata:
  name: arble-runtime
  namespace: arble
spec:
  selector: {app: arble-runtime}
  ports:
    - {name: http, port: 8080, targetPort: http}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: arble
  namespace: arble
  annotations:
    # Streaming must not be buffered or cut off mid-run.
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
spec:
  ingressClassName: nginx
  tls:
    - hosts: [arble.internal.example.com]
      secretName: arble-tls
  rules:
    - host: arble.internal.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: arble-gateway
                port: {name: http}

Persistent volumes

Only arble-memory needs a PersistentVolume. Use ReadWriteOnce on block storage — the vector index is not safe to share across Pods.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: arble-memory
  namespace: arble
spec:
  serviceName: arble-memory
  replicas: 1
  selector:
    matchLabels: {app: arble-memory}
  template:
    metadata:
      labels: {app: arble-memory}
    spec:
      containers:
        - name: memory
          image: ghcr.io/arble-ai/memory:1.8.2
          volumeMounts:
            - {name: index, mountPath: /var/lib/arble/index}
  volumeClaimTemplates:
    - metadata: {name: index}
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: fast-nvme
        resources: {requests: {storage: 100Gi}}

Secrets

Never put credentials in a ConfigMap or a Helm values file committed to git. Reference existing Secrets, ideally projected from an external manager:

kubectl create secret generic arble-postgres \
  --namespace arble \
  --from-literal=url='postgres://arble:...@pg.internal:5432/arble'

Autoscaling

Scale the runtime on concurrent runs, not CPU — a run waiting on a model uses almost no CPU while still consuming a slot.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: arble-runtime
  namespace: arble
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: arble-runtime
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric: {name: arble_active_runs}
        target: {type: AverageValue, averageValue: "15"}
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

Workers scale on queue depth (arble_tool_queue_depth). Both metrics are exported on :9090 and require an adapter such as prometheus-adapter.

Node pools and rolling updates

Separate pools by workload shape. Arble components are CPU workloads and should never be scheduled onto GPU nodes, where they waste expensive capacity.

      nodeSelector:
        arble.ai/pool: runtime
      tolerations:
        - key: arble.ai/dedicated
          operator: Equal
          value: runtime
          effect: NoSchedule

Taint GPU nodes for inference only, and let the model backend be the sole workload tolerating that taint.

maxUnavailable: 0 with maxSurge: 1 brings up a new Pod before removing an old one. Combined with the preStop drain hook and a terminationGracePeriodSeconds longer than the drain timeout, in-flight runs complete rather than failing. Add a PodDisruptionBudget so node maintenance can’t take the service down:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: arble-runtime
  namespace: arble
spec:
  minAvailable: 2
  selector:
    matchLabels: {app: arble-runtime}

Models

Arble does not ship a model. It speaks the OpenAI chat-completions protocol to whatever backend you configure, which is why local and hosted providers are interchangeable.

models:
  default: local-qwen
  embedding: local-embed
  providers:
    - name: local-qwen
      kind: openai_compatible
      base_url: http://vllm.models.svc.cluster.local:8000/v1
      model: Qwen3-32B-Instruct
      max_concurrency: 32

    - name: local-embed
      kind: openai_compatible
      base_url: http://tei.models.svc.cluster.local:8080/v1
      model: bge-large-en-v1.5

    - name: anthropic-fallback
      kind: anthropic
      api_key_secret: arble-anthropic
      model: claude-sonnet-4-5

Routing and fallback

Route by task class rather than pinning one model everywhere. Planning benefits from a stronger model; summarization and memory compaction do not.

A fallback chain is evaluated in order on connection failure, 5xx, or timeout. It is not evaluated on a 4xx — a malformed request will fail the same way everywhere.

models:
  routes:
    planner:
      primary: local-qwen
      fallback: [local-qwen-8b, anthropic-fallback]
      timeout: 60s
    summarizer: local-qwen-8b
    embedding: local-embed

In air-gapped deployments, omit hosted providers entirely. A configured-but-unreachable provider adds a timeout to every failure path for no benefit.

Load balancing

For multiple replicas of one model, put a Service or load balancer in front and give Arble a single base_url. Arble does not implement inference-aware load balancing, and a naive round-robin in front of vLLM will degrade prefix-cache hit rates.

Set max_concurrency to match what the backend can serve. Arble queues beyond it rather than overwhelming the GPU and triggering timeouts across every in-flight run.

Storage

Four storage systems, each with a distinct role. Only Postgres and the object store hold data you cannot rebuild.

SQLite is the default for desktop and single-node deployments. It is genuinely production-capable for one node and one process, and unsuitable for anything with multiple runtime replicas. Migrate before you scale out, not during.

PostgreSQL 14+ is required for multi-node. Use 16, and enable pgvector if you want the vector index in Postgres rather than a separate service:

database:
  url: postgres://arble@pg.internal:5432/arble
  max_connections: 40
  vector_backend: pgvector

Size the connection pool as max_connections × runtime replicas and check it against the server’s limit. Exhausting Postgres connections is the most common cause of a cluster-wide stall.

Object storage is any S3-compatible endpoint — AWS S3, MinIO, Ceph RGW, Cloudflare R2. Enable object versioning; it turns an accidental deletion into a recoverable event.

Redis 6.2+ handles queues and coordination. Enable AOF persistence so a restart doesn’t drop queued work. Use Sentinel or Cluster for HA.

Retention

Retention is configured per data class. Defaults are conservative; tighten them to match your compliance posture.

retention:
  runs: 90d
  messages: 90d
  audit_log: 365d
  events: 30d
  files: 180d
  memory: indefinite

A nightly job enforces retention and vacuums reclaimed space. Audit log retention should meet or exceed your longest compliance window — it is the record you will be asked for.

Memory

Memory is what makes an agent useful on the second run. It is also the component with the most sensitive contents, since it accumulates facts about your systems over time.

Short-term memory is session context: messages and tool results in the current conversation. It lives in Postgres, is bounded by the model’s context window, and is discarded when the session is deleted.

Long-term memory is durable per-project facts, deliberately written and semantically retrieved. It outlives sessions by design — deleting a session does not delete what it learned.

Embeddings are generated by the configured embedding model and stored in the vector index. Changing embedding models requires a full reindex, since vectors from different models are not comparable.

Compression collapses related entries into fewer, denser ones. Run it on a schedule for long-lived projects — an unbounded memory costs context on every single run.

arble memory reindex --model local-embed-v2
arble memory summarize --older-than 90d

Retention for memory defaults to indefinite, which is usually correct. Facts don’t expire on a timer; prune them when they stop being true.

Encryption applies at rest through your storage layer, and Arble additionally encrypts memory contents with a project-scoped key before writing. A database dump does not yield readable memory without the key:

memory:
  encryption:
    enabled: true
    key_source: vault
    vault_path: secret/arble/memory-key

Networking

Arble needs exactly one inbound port. Everything else is internal, and treating it that way is most of your security posture.

HTTPS and TLS. TLS 1.2 minimum, 1.3 preferred. Terminate at the load balancer or ingress. Plain HTTP is rejected rather than redirected — a redirect leaks the bearer token in the first request.

Reverse proxy. Any proxy works, with two required settings: response buffering off, and a read timeout longer than your longest run. Both defaults are wrong for streaming.

location / {
    proxy_pass http://arble-gateway:8080;
    proxy_http_version 1.1;
    proxy_buffering off;
    proxy_read_timeout 3600s;
    proxy_set_header X-Forwarded-Proto https;
}

Load balancer. Runs are not pinned to a replica, so no session affinity is required. Health checks should target /readyz, not /healthz — the former knows whether dependencies are reachable.

Internal networking. Runtime-to-worker is gRPC on 7070, and worker-to-anything-else should be denied by default. Workers execute tool code; they are the component most likely to be attacked.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: arble-worker-egress
  namespace: arble
spec:
  podSelector:
    matchLabels: {app: arble-worker}
  policyTypes: [Egress]
  egress:
    # DNS, and explicitly allowed destinations only.
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports: [{protocol: UDP, port: 53}]
    - to:
        - podSelector:
            matchLabels: {app: vllm}
      ports: [{protocol: TCP, port: 8000}]

Private DNS. Use internal names for every service-to-service hop, and make external_url match the certificate exactly. A mismatch breaks OAuth redirects and webhook signature verification in ways that are tedious to diagnose.

Firewall. Allow 443 inbound to the load balancer. Deny everything else inbound. Outbound egress should be an explicit allowlist — model endpoints, your registry, and any remote MCP servers you have approved.

Air-gapped deployments. Mirror images to an internal registry, use local models only, and disable telemetry and the connector directory. Arble runs with no egress; the CLI warns once at startup that update checks are unavailable.

telemetry:
  enabled: false
registry:
  connector_directory: disabled
  mcp_registry: disabled
updates:
  check: false

Authentication

Self-hosted Arble authenticates users through your identity provider and services through API keys. There is no separate Arble user directory to maintain.

OIDC is the recommended integration and works with Okta, Entra ID, Auth0, Keycloak and Google Workspace:

auth:
  oidc:
    issuer: https://login.example.com
    client_id: arble-production
    client_secret_secret: arble-oidc
    redirect_url: https://arble.internal.example.com/auth/callback
    scopes: [openid, profile, email, groups]

  role_mapping:
    "arble-admins": admin
    "platform-engineering": maintainer
    "engineering": member

SAML is available for providers without OIDC. It supports SP-initiated flow and signed assertions; IdP-initiated flow is not supported, since it can’t carry the state parameter Arble uses to prevent login CSRF.

LDAP authenticates against Active Directory or OpenLDAP directly. Prefer OIDC where you have the option — LDAP requires Arble to handle credentials, which OIDC avoids entirely.

Group mapping drives authorization. Map IdP groups to Arble roles so access changes in one place.

API keys authenticate services, scoped to a project or organization. Prefer project scope — a leaked key then reaches one project’s data, not everything.

Service accounts are non-human identities for CI and automation. They hold their own permission grants and appear in the audit log under their own name, which is what makes an unattended action attributable.

arble service-accounts create ci-pr-review \
  --project api-gateway \
  --permissions "network.http:api.github.com"

Organizations partition projects, users and billing. In self-hosted deployments most operators run one organization; use several when you need hard tenant separation with no shared memory or connectors.

Secrets

Arble reads secrets from a configurable source and never writes them to disk in plaintext. Every value marked secret is redacted from logs, traces and audit entries.

Environment variables are acceptable for evaluation and single-host Docker. They are visible in docker inspect and in a process listing, so avoid them in production.

HashiCorp Vault is the recommended production source, with either token or Kubernetes auth:

secrets:
  backend: vault
  vault:
    address: https://vault.internal.example.com
    auth_method: kubernetes
    role: arble
    mount: secret/arble

Kubernetes Secrets are the practical default in-cluster. Enable encryption at rest for etcd, or project from an external manager with the Secrets Store CSI driver — a plain Secret is base64, not encryption.

Docker Secrets work in Swarm, mounted at /run/secrets. In plain Docker Compose, use a read-only mounted file rather than an environment variable.

Rotation is non-disruptive when done in the right order. Create the new value, deploy it, verify, then revoke the old one:

arble api-keys create --name api-gateway-prod-2 --project api-gateway
# deploy, verify, then:
arble api-keys revoke key_8fRt2Ls

Rotate the memory encryption key with arble memory rotate-key, which re-encrypts in place. Do not rotate it by hand; a partially re-encrypted store is not recoverable without both keys.

Scaling

Scale the plane that is actually saturated. Adding runtime replicas will not help if the bottleneck is GPU throughput, and it may make it worse.

Horizontal scaling applies to the gateway, runtime and workers — all stateless. This is the first lever for almost every capacity problem.

Vertical scaling applies to arble-memory and Postgres. The vector index is single-writer; give it more CPU, RAM and faster disk rather than more replicas.

Queue workers execute scheduled and triggered runs. Scale them on arble_job_queue_depth, separately from interactive traffic, so a nightly batch doesn’t starve users.

GPU pools scale independently of Arble. Keep them tainted and dedicated, and size on tokens per second rather than request count — a long summarization consumes far more than a short planning step.

Memory workers handle embedding and compaction asynchronously. If memory writes lag behind runs, add memory workers before scaling the index.

Session workers maintain long-lived streaming connections. Each holds an open connection, so they are memory- and file-descriptor-bound rather than CPU-bound. Raise ulimit -n accordingly.

High availability

HA means surviving the loss of one node without dropping requests. Three things must be true: no singleton on the request path, a failover story for state, and health checks that tell the truth.

Multiple replicas. Run 3+ gateway and runtime replicas across distinct nodes. Use anti-affinity so a single node failure can’t take all of them:

      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels: {app: arble-runtime}
              topologyKey: kubernetes.io/hostname

Leader election. Scheduled jobs, retention and migrations must run exactly once. Replicas elect a leader through Redis, and only the leader executes them — no cron sidecar or manual designation is required.

Database failover. Use managed Postgres with automated failover, or Patroni if self-managing. Arble reconnects with backoff and reports unready while the primary is unavailable; runs in flight fail and are retryable.

Health checks. /healthz for liveness, /readyz for readiness and load-balancer membership. Never point a liveness probe at /readyz — a brief database blip would restart every Pod simultaneously and turn a degradation into an outage.

Automatic restart. Kubernetes handles this through the liveness probe. On Docker, use restart: unless-stopped. On bare metal, a systemd unit with Restart=always and RestartSec=5.

Monitoring

Every component exports Prometheus metrics on :9090/metrics and structured JSON logs on stdout.

scrape_configs:
  - job_name: arble
    kubernetes_sd_configs:
      - {role: pod, namespaces: {names: [arble]}}
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: "arble-.*"
        action: keep

The metrics worth alerting on:

Logs are JSON with a request_id, session_id, run_id and agent_id on every line. Ship them with any collector; the fields are designed to correlate a user report to a run.

{"ts":"2026-07-30T14:02:19Z","level":"info",
 "msg":"tool call completed",
 "request_id":"req_5Kp2Nvx","run_id":"run_2Kd8vQx",
 "tool":"github.create_pull_request",
 "decision":"approved","duration_ms":212}

Tracing uses OpenTelemetry. A trace spans the gateway, runtime, planner, each tool call and each model request — which is what tells you whether a slow run was the model or a tool.

telemetry:
  otlp_endpoint: http://otel-collector.observability:4317
  sample_rate: 0.1

Sample at 10% in steady state. Trace volume scales with tool calls, not requests, and a single complex run can emit dozens of spans.

Health endpoints. /healthz liveness, /readyz readiness with dependency checks, /version build and migration state. All three are unauthenticated and safe to expose internally.

Security

Arble executes model-generated tool calls against your systems. Treat the worker as hostile-adjacent and the permission gate as the control that matters.

permissions:
  default: never_allow
  grants:
    - {capability: filesystem.read, scope: "/workspace"}
    - {capability: network.http, scope: "api.github.com"}

Backups

Back up three things. Two are small, and the third is where the data actually is.

Database. Logical dumps are simple and slow to restore; WAL archiving gives point-in-time recovery and is what you want in production:

pg_dump --format=custom --compress=9 \
  "$ARBLE_DATABASE_URL" > arble-$(date +%F).dump

Memory metadata is in Postgres and the vector index is derived, so the database backup covers it. Restore, then run arble memory reindex.

Configuration belongs in git — arble.yaml, Helm values, manifests. Never commit secrets; reference them.

Snapshots. Volume snapshots are convenient but are not a consistent database backup unless the filesystem supports atomic snapshots and you quiesce writes. Use them alongside pg_dump, not instead of it.

Restore into an empty database, then start the runtime — it will detect the schema version and refuse to start if it mismatches the binary, which is the behavior you want:

pg_restore --dbname="$ARBLE_DATABASE_URL" --clean arble-2026-07-30.dump
arble server migrate
arble memory reindex

Version history. Configuration changes, permission grants and skill versions are recorded, so you can see what changed before an incident: arble audit config --since 7d.

Disaster recovery

Define your targets first. RPO is how much data you can lose; RTO is how long you can be down. Every choice below follows from those two numbers.

Cold backup. Nightly dumps in object storage, infrastructure as code, nothing running. Restore by applying manifests and loading the dump. Appropriate for internal tooling.

Warm standby. A second environment deployed and current, scaled to zero or minimal replicas, with continuous WAL shipping. Failover is a scale-up and a DNS change.

Hot standby. Full capacity in a second region with streaming replication and a global load balancer. Justified when Arble is on a critical path.

Replication. Postgres streaming replication for the control plane, cross-region bucket replication for objects, and Redis replication only if you care about in-flight runs — usually you don’t, since they’re retryable.

Failover is four steps, and the order matters: promote the database, point configuration at the new primary, scale up the runtime, move DNS.

arble server drain --timeout 120s     # if the old site is reachable
# promote replica, update the database secret, then:
kubectl -n arble rollout restart deployment/arble-runtime
arble server verify --all

What is not recoverable: memory encrypted with a key you didn’t back up, and object-store contents with versioning disabled and no replication. Verify both are covered before you need them.

Updating

Arble uses semantic versioning. Patch and minor releases are backward compatible; major releases may require migration steps documented in the release notes.

Version compatibility. The gateway, runtime, worker and memory components must share the same minor version. Mixing minors is unsupported and the runtime logs a warning and refuses some operations.

Database schema is versioned independently. A binary refuses to start against a schema newer than it understands, which is what makes rollback safe.

Rolling updates. Run migrations first, then roll the Deployments. Migrations are additive within a minor release, so the old version keeps running against the new schema during the rollout:

kubectl -n arble create job arble-migrate-1-8-2 \
  --image=ghcr.io/arble-ai/runtime:1.8.2 -- arble server migrate

helm upgrade arble oci://ghcr.io/arble-ai/charts/arble \
  --version 1.8.2 --namespace arble --values values.yaml --wait

Zero downtime requires maxUnavailable: 0, working readiness probes, the preStop drain hook, and a PodDisruptionBudget. With all four, in-flight runs complete on the old replicas while new traffic goes to the new ones.

Rollback. Roll the application back first; only roll the schema back if the release notes say the migration was destructive. Because migrations are additive within a minor, an application rollback is safe without a schema rollback:

helm rollback arble --namespace arble

Always test an upgrade against a staging environment restored from a production backup. A migration’s duration is a function of your data volume, and finding that out in production is avoidable.

Troubleshooting

Start with arble server verify --all, which checks database, Redis, object store, model backends and sandbox in one pass.

arble server verify --all
arble mcp health
arble connectors status github
arble permissions audit --agent agt_5wTn9Kd --since 1h
arble admin storage-usage
arble admin retention apply --dry-run

kubectl -n models logs deploy/vllm --tail=50
kubectl -n arble get ingress arble -o yaml | grep -A5 annotations

Best practices

1. Pin exact image versions in production. Never deploy latest.
2. Keep arble.yaml, Helm values and manifests in version control; reference secrets, never inline them.
3. Run migrations as a discrete step before rolling the application, not as a container entrypoint.
4. Use managed Postgres with automated failover unless you already operate Patroni well.
5. Point liveness at /healthz and readiness at /readyz. Never point liveness at a dependency check.
6. Set maxUnavailable: 0, a preStop drain hook, and a terminationGracePeriodSeconds above the drain timeout.
7. Add a PodDisruptionBudget so node maintenance can’t drain the service to zero.
8. Turn off proxy response buffering and raise the read timeout before debugging streaming.
9. Autoscale the runtime on arble_active_runs, not CPU — a run waiting on a model is idle.
10. Taint GPU nodes for inference only; never schedule Arble components onto them.
11. Deny worker egress by default and allowlist explicitly. This is your highest-value control.
12. Set permissions.default: never_allow and grant capabilities narrowly, with scopes.
13. Use project-scoped tokens and service accounts so unattended actions are attributable.
14. Integrate OIDC and map IdP groups to roles, so access changes in one place.
15. Back up the memory encryption key separately from the database.
16. Enable object versioning on the bucket; it makes deletion recoverable.
17. Restore a backup into staging on a schedule and time it. Measure RTO, don’t assume it.
18. Ship audit logs off-host, and retain them at least as long as your compliance window.
19. Run arble memory summarize periodically; unbounded memory costs context on every run.
20. Test upgrades against staging restored from a production backup, so migration duration is known.

Reference

• Docker — images, volumes, ports, Compose
• Kubernetes — Helm chart, manifests, autoscaling, node pools
• Configuration — arble.yaml structure and precedence
• Environment variables — full list via arble server config --list
• Networking — TLS, proxy settings, NetworkPolicy, air-gapped
• Storage — Postgres, Redis, object storage, retention
• Authentication — OIDC, SAML, LDAP, service accounts
• Scaling — metrics, bottleneck table, worker pools
• Security — sandbox, permissions, audit, isolation
• Monitoring — metrics, logs, tracing, health endpoints
• Backups — database, objects, keys, restore procedure
• CLI — arble server and arble admin commands
• API Reference — REST endpoints exposed by your deployment

FAQ