Deployment

memoturn is self-hostable with no cloud lock-in. Dev uses Docker Compose; production is the same components (managed or self-run).

Local / dev dependencies

bun run infra:up     # Postgres, Apache Doris (FE + BE), Redis/Valkey, MinIO (infra/docker-compose.dev.yml)
bun run infra:down
bun run infra:logs

With TELEMETRY_ENGINE=postgres in .env, infra:up skips the Doris containers entirely (they sit behind a compose profile) — see Telemetry engine.

Telemetry engine: Doris or Postgres

memoturn runs its telemetry store on one of two engines, selected by TELEMETRY_ENGINE (ADR-0002). Both pass the same behavioral conformance suite; the product is identical on either.

	doris (default)	postgres
Best for	Sustained volume, long retention, big scans	Small installs — the lightest possible footprint
Telemetry lives in	Apache Doris FE + BE	The same Postgres you already run (schema telemetry)
Extra containers	2 (JVM-based, ~4 GB+ between them)	None (Postgres image must include pgvector — the shipped compose files use pgvector/pgvector:pg16)
Comfortable up to	See Doris sizing	Roughly 10–50 M observation rows / low hundreds of GB
Vector search	Exact cosine scan	Exact cosine scan (pgvector)

Selecting the engine:

• Dev: set TELEMETRY_ENGINE=postgres in .env — bun run infra:up, infra:wait, and bun run db:telemetry all follow it (Doris containers never start; migrations from infra/postgres-telemetry/ apply into the telemetry schema).

• Production: layer the overlay on the single-VM stack —

  docker compose -f infra/docker-compose.prod.yml \
                 -f infra/docker-compose.prod.postgres.yml up -d --build

  The overlay parks the Doris services (never started), rewires service dependencies, and sets TELEMETRY_ENGINE=postgres for the API, worker, and migrate step. Requires Docker Compose ≥ 2.24. .env must still define DORIS_PASSWORD (any placeholder — the base file’s variable interpolation runs before the overlay merges; nothing Doris-related runs).

When to graduate to Doris: trace-list facets and dashboards getting slow, sustained ingest in the thousands of rows/sec, long retention at high volume, or embedding spaces past ~100k vectors. The move is a defined, verifiable runbook — a seam-level row copy with no API downtime and blob replay as the fallback (ADR-0004); it also works in reverse (within the Postgres envelope) for downsizing. Retention and deletes behave identically on both engines.

Full self-host stack

infra/docker-compose.yml builds and runs the API + worker alongside all dependencies:

docker compose -f infra/docker-compose.yml up -d --build

Images are built from docker/{api,worker,console}.Dockerfile (all oven/bun). The console is a static SPA — build it (bun --filter @memoturn/console build) and serve the output behind any static host / CDN, with a reverse proxy routing /api/* to the API and SPA-fallback (rewrite unknown paths to index.html) for deep links.

Single-VM production (Docker Compose + Caddy)

infra/docker-compose.prod.yml is a self-contained, HTTPS-terminated stack for one server: Caddy (auto Let’s Encrypt) in front of the console + API + worker, plus self-hosted Postgres, Doris (FE + BE), Valkey, and MinIO. Caddy is the only published port (80/443); everything else stays on the internal network. A one-shot doris-setup service sets the Doris root password (DORIS_PASSWORD is required in production), and a one-shot migrate service runs the Prisma + Doris migrations before the API/worker start.

Never expose Doris ports without a root password. While root’s password is empty, the FE HTTP query API (8030) accepts any credentials — anyone who can reach it can run SQL. The bundled prod compose keeps Doris internal-only and always sets a password; the dev compose binds 9030/8030 to loopback because dev runs password-less.

Prerequisites: a VM with Docker, a DNS A/AAAA record pointing at it, and ports 80+443 open.

cp .env.production.example .env          # set MEMOTURN_DOMAIN, ACME_EMAIL, and the secrets
openssl rand -base64 48                  # generate each secret (BETTER_AUTH_SECRET, ENCRYPTION_KEY, passwords)
bun run prod:up                          # docker compose -f infra/docker-compose.prod.yml --env-file .env up -d --build

Companion scripts: bun run prod:ps (status), bun run prod:logs (tail logs), bun run prod:down (stop).

The compose file derives AUTH_BASE_URL=https://DOMAIN/api and AUTH_TRUSTED_ORIGINS=https://DOMAIN from MEMOTURN_DOMAIN; required secrets use ${VAR:?} so a missing value aborts the command. Caddy routes /api/* to the API (prefix stripped, matching the dev Vite proxy) and everything else to the console SPA — so the console’s default VITE_API_BASE=/api works unchanged.

First admin: do not run bun run seed in production (it seeds the well-known dev key). Open https://DOMAIN/, sign up the first admin (Better Auth email/password) — the org plugin auto-provisions a default project — then mint an SDK API key from Settings. Point the SDK at https://DOMAIN/api.

Backups: bun run prod:backup (scripts/backup.sh) dumps Postgres (pg_dump | gzip) and mirrors the blob bucket — the replayable raw event log, the source of truth from which Doris can be rebuilt — into ./backups/, keeping the newest BACKUP_KEEP (default 7) of each. Schedule it with cron (0 2 * * * cd /opt/memoturn && bun run prod:backup) and ship ./backups/ off-host; a backup on the same disk is not a backup. Doris itself is not snapshotted — recovery is blob replay (add Doris BACKUP SNAPSHOT to an S3 repository if you need faster restores). Restore commands are documented at the top of the script. All datastores persist to named volumes (pgdata, dorisfemeta, dorisbedata, redisdata, miniodata).

Note: a single VM has no HA. If volume or uptime needs grow, move to the Helm chart below with managed datastores.

Kubernetes (Helm)

For production, the chart at infra/helm/memoturn deploys the stateless API (behind an HPA), the worker, and the console; Postgres / Doris / Redis / blob are expected to be external (managed services or operators — e.g. the community doris-operator or any managed Doris; memoturn only needs the FE MySQL endpoint). A pre-install/upgrade hook Job runs the Prisma + Doris migrations (bun run db:migrate && bun packages/telemetry/src/migrate.ts) before pods roll. Published images come from ghcr.io/memoturn/*.

helm install memoturn ./infra/helm/memoturn -f my-values.yaml

See the chart README for required values (datastore URLs, betterAuthSecret, encryptionKey) and the ingress / autoscaling options.

Migrations

bun run db:migrate      # Prisma (Postgres)
bun run db:telemetry    # Doris DDL (infra/doris/*.sql, tracked in a schema_migrations ledger)

Run these on deploy. After a Prisma schema change, the client is regenerated by postinstall (or bun run db:generate).

Upgrading

The compose stacks build from source, so an upgrade is: pin the new version, rebuild, and let the one-shot migrate service run before the app services start.

bun run prod:backup                      # snapshot Postgres + the blob event log first
git fetch --tags && git checkout vX.Y.Z  # pin a release tag (or a commit) — don't track main in prod
bun run prod:up                          # rebuilds images and applies the upgrade

docker compose up -d --build (what bun run prod:up runs) handles the ordering for you:

1. The one-shot migrate service runs bun --filter @memoturn/db migrate (Prisma) and bun packages/telemetry/src/migrate.ts — the latter applies any new Doris DDL from infra/doris/*.sql, tracked in the schema_migrations ledger, so Doris schema changes ride the same step. Nothing else to run by hand.
2. api and worker start only after migrate exits successfully (service_completed_successfully) — they never run against an unmigrated schema.
3. console restarts alongside them; it’s a static SPA, so order doesn’t matter beyond Caddy staying up (Caddy and the datastores keep running through the upgrade — expect a brief API blip while the api container swaps).

If you deploy prebuilt images (ghcr.io/memoturn/*) instead of building, pin the image tag, docker compose pull, then up -d — same migrate-first ordering applies.

Kubernetes: the Helm chart runs the same migrations in a pre-install,pre-upgrade hook Job (templates/migrate-job.yaml), so helm upgrade with a new image tag migrates before pods roll.

Rollback expectations: migrations are forward-only — there are no down migrations. Roll the application back to the previous tag only if no migration shipped in between; otherwise restore Postgres from the pre-upgrade backup. Doris is rebuildable from the blob raw-event log (the replay path), so telemetry is never the thing that blocks a recovery.

Scaling

• API is stateless — scale horizontally behind a load balancer.
• Worker scales by process count and WORKER_CONCURRENCY; the BullMQ queue distributes ingest jobs. It also serves /health + /metrics on WORKER_PORT (default 3002) for liveness/observability probes.
• Apache Doris holds the high-volume telemetry; use a managed Doris or the community doris-operator in production — the store only needs the FE MySQL endpoint. Postgres stays small (metadata).
• The raw blob event log is the source of truth — Doris can be rebuilt from it.

Doris sizing

Starting points, not guarantees — telemetry width (payload sizes, evaluator volume, retention window) moves these numbers a lot. Observe and adjust; the failure mode of an undersized BE is analytical queries (and eventually ingest inserts) erroring with MEM_ALLOC_FAILED.

Load	Observations/day	DORIS_FE_XMX	DORIS_BE_MEM_LIMIT	CPU (BE)	Disk	WORKER_CONCURRENCY
Light	< 100k	2048m	4G	2–4 vCPU	50 GB SSD	10 (default)
Medium	~1M	4096m (default)	8G–12G	4–8 vCPU	200–500 GB SSD	20, consider TELEMETRY_STREAM_LOAD=true
Heavy	> 5M	8192m	24G+ (dedicated host)	16+ vCPU	1 TB+ NVMe	multiple worker replicas + Stream Load

Notes:

• On the shared single-VM stack, DORIS_BE_MEM_LIMIT defaults to 40% of host memory (dev compose: 6G) — an absolute value is safer once other services compete for the same host.
• The compose stacks run one FE + one BE with no HA — fine well into the medium tier, but heavy sustained volume (or uptime requirements) is the signal to move to the Helm chart with a managed Doris or the doris-operator.
• Postgres and Redis stay small at every tier (metadata + queues); size the blob bucket for the raw event log, which grows with retention, not query load.

Data retention

Set a per-project max age; a daily worker cron deletes older telemetry:

curl -u pk-mt-dev:sk-mt-dev -X POST http://localhost:3001/v1/retention \
  -H 'content-type: application/json' -d '{"days":90}'      # 0 = keep forever
curl -u pk-mt-dev:sk-mt-dev -X POST http://localhost:3001/v1/retention/apply   # run now

Production checklist

• Set a strong BETTER_AUTH_SECRET (32+ chars) and a distinct ENCRYPTION_KEY.
• Use managed/persistent Postgres, Doris, Redis, and S3 (rotate the dev credentials; set DORIS_PASSWORD).
• Put the API and console behind TLS; set AUTH_BASE_URL / AUTH_TRUSTED_ORIGINS.
• Configure object storage (S3/R2/GCS) for the blob event log + exports.
• Optionally cap ingest with RATE_LIMIT_PER_MINUTE (per-project global limit; 0 disables it).
• For enterprise tenants, register an SSO provider (OIDC/SAML) per organization from the Organizations page, and configure the event sink (CDP forwarding) / PII masking per project as needed.
• Walk the security hardening guide — a checklist of every security knob (proxy trust, rate limits, metrics exposure, SSRF policy, account hardening) with defaults and failure modes.

See Configuration for the full variable reference.