Architecture¶

Overview¶

Gradient is a multi-crate Rust workspace with two separate binaries: gradient (the server) and gradient-worker (the build worker).

Server¶

The server binary is one process whose long-lived work runs under a supervision tree (gradient_util::supervision, on ractor). The tree is started on first use from the shared Shutdown coordinator; every subsystem registers its loops as children with Shutdown::supervise.

root
├── graph                                actor: sole writer of the dependency graph
│                                        and the cache index; stopped last
├── scheduler                            supervisor node
│   ├── scheduler-core                   actor: WorkerPool + JobTracker behind messages
│   ├── trigger-dispatch, eval-dispatch  periodic passes (5s)
│   └── build-dispatch                   actor: 5s tick plus coalesced kicks
├── sessions                             supervisor: one actor per worker connection
├── worker-sample, instance-metrics      periodic passes
├── worker-liveness, graph-consistency   periodic passes, absent when disabled
├── cache-maintenance, sign-sweep,
│   debug-index, eval-cache-sweep        cache sweeps
├── retention, rollup, otlp-snapshot     metrics pipeline
└── outbound-connect                     dials workers with a registered URL

The scheduler's state (WorkerPool, JobTracker) is private to one actor; Scheduler is a facade whose every method is one message with a typed reply, so a claim, a release or a peer revocation is atomic and nothing holds a lock across a database call. The scheduler pushes to a session only through a SessionPort signal (Offers, Reauth, Abort, Drain); a burst of enqueues collapses into one offer per generation. Each worker connection is a SessionActor: its reader task delivers one inbound frame at a time with a call and reads the next only after the reply, so TCP backpressure holds. Order-independent RPCs (CacheQuery, QueryKnownDerivations, WorkerMetrics) still run as tracked tasks off the session. When the core actor is respawned, the sessions supervisor re-registers every live session together with the jobs it still runs.

The graph actor (gradient-graph) is the only code that writes derivation, derivation_build, derivation_dependency, derivation_output, derivation_input_source, build_job, build_attempt, cached_path and cached_path_reference. Sessions, the scheduler, the web handlers and the cache sweeps reach it through state.graph: an evaluation batch, a NAR commit, an anchor transition, a requeue or a demotion is one message and one transaction, so two workers walking overlapping graphs can no longer race each other on a derivation row or on an edge. Batches queued at the same moment are written in one transaction with a savepoint each, and a worker's known-derivations query is answered only after the batches queued before it, so it never prunes a subtree whose edges are still unwritten. Reads that need no ordering (CacheQuery, board and API queries) stay on the pools. Effects that leave the process (forge reports, notifications) are still spawned after the write; #597 moves them into an outbox. Startup recovery, the maintenance deletions (Gc, #597) and the debug indexer's flag stay outside the actor.

A child that panics or exits unexpectedly is respawned after an exponential backoff (1s doubling to 60s, reset after five healthy minutes); a pass that runs past its budget is cancelled in place and ticks again. Restarts, pass errors, budget timeouts and the last successful pass are reported per child on /board/health. Everything that outlives one request but is not a loop runs as a tracked task so shutdown drains it; bare tokio::spawn is a clippy error in the server crates. Axum serves the HTTP API and the worker WebSocket; the scheduler, cache and CI code are libraries the tree drives.

Worker¶

gradient-worker is a standalone process that connects to the server over WebSocket at /proto. It handles fetch, eval, build, and sign tasks dispatched by the server's scheduler. Workers can run co-located on the server host or on separate machines.

Server has no Nix daemon¶

The server does not require access to a Nix daemon. All store interaction (eval, build, fetch, sign) happens on worker nodes, which talk to a local nix-daemon via harmonia. The server reads .narinfo responses straight from DB rows (cached_path) and serves artefact downloads from nar_storage (local FS or S3), extracting individual files from the stored NARs on the fly. GC roots are created and removed only on the worker that builds the output.

Crates¶

Every crate is backend/gradient-<name>; the workspace root is the gradient-server binary.

core           AppState, config, bootstrap, upstream probe
db             every query and the graph reconciler, over the pools
entity         SeaORM entities, one module per table
migration      SeaORM migrator
graph          the graph actor: every write to the graph and the cache index
scheduler      worker pool, job tracker, dispatch loops
proto          worker protocol: sessions, NAR transfer, dispatch
web            Axum HTTP API and the binary cache endpoints
cache          cache sweeps: maintenance, signing, debug index, retention
ci             evaluation triggers, forge checks, declarative apply
forge          per-forge reporters, webhook parsing, signature checks
state          declarative state DTOs and apply
storage        NAR and log storage (local FS or S3)
worker         gradient-worker binary (fetch, eval, build, sign)
eval           standalone flake evaluator used by the worker
nix, sources   nix bindings and store-path/source helpers
exec, util     process execution, shutdown, supervision, HTTP clients
score          scoring policies for job assignment
notify         email and notification senders
report         evaluation snapshots for support reports
flake-lock     native `flake.lock` model and updater
types          shared ids, wire types, runtime config
test-support   shared test fixtures for the workspace

core¶

Defines ServerState - the shared Arc<ServerState> threaded through every Axum handler and every spawned task. It holds:

• db: DatabaseConnection - SeaORM PostgreSQL pool
• cli: Cli - resolved configuration from env/flags

Key modules: core::executor (Nix store interaction), core::sources (key generation, NAR helpers), core::input (validation), core::database (common queries).

entity¶

One module per database table using SeaORM derive macros. The naming convention is:

Key entities and their relationships:

project
  ├── worker_registration[]    registered worker auth tokens
  ├── cache[]                  binary caches (via subscription)
  ├── derivation[]             immutable per-project "what to build" records
  │     ├── derivation_output[]         (one per Nix output: out, dev, doc, ...)
  │     ├── derivation_dependency[]     (edges: derivation → dependency)
  │     └── derivation_feature[]
  └── task[]
        └── evaluation[]
              ├── commit
              ├── build[]                (one per attempt at a derivation)
              └── entry_point[]          (top-level builds for this eval)

The build row is the "attempt" - it carries status, log_id, and build_time_ms. Everything immutable about the derivation (path, architecture, outputs, dep graph, required features) lives on derivation and is shared across every evaluation that touches it. A rebuild on failure inserts a new build row on the same derivation.

derivation_dependency is a directed edge table: derivation → dependency means the dependency derivation must be built before derivation. The graph is stored once per derivation, not once per evaluation.

cache_derivation (cache, derivation) records that a cache holds the complete closure of a derivation. The cacher only inserts a row once every output of the derivation is is_cached = true AND every transitive dependency already has a matching cache_derivation row for the same cache.

worker_registration stores (peer_id, worker_id, token_hash) - the challenge-response auth tokens issued when a peer (project, cache, or proxy) registers a worker.

builder¶

Manages the evaluation and build queues. Jobs are dispatched to proto workers via the Scheduler. The builder no longer runs builds directly - it enqueues PendingEvalJob and PendingBuildJob entries that the proto scheduler delivers to connected workers.

See Internals for algorithm details.

proto¶

Handles the WebSocket /proto endpoint and the scheduler that dispatches jobs to connected workers:

• handler.rs - WebSocket lifecycle: handshake, challenge-response auth, capability negotiation, job dispatch loop
• scheduler/ - WorkerPool tracks connected workers; JobTracker tracks pending and active jobs; dispatch loops push JobOffers to eligible workers
• messages/ - rkyv-serialized wire message types (ServerMessage, ClientMessage)

The scheduler is injected into the Axum router as Extension<Arc<Scheduler>> and shared with the builder.

worker¶

The gradient-worker binary. Connects to the server over WebSocket, performs the challenge-response handshake, and executes dispatched jobs:

• executor/eval.rs - Nix flake evaluation (spawns evaluator subprocesses)
• executor/build.rs - Nix store builds via the local daemon
• handshake.rs - client-side challenge-response auth
• config.rs - WorkerConfig parsed from env vars / CLI args

web¶

Axum HTTP server. All API routes live under /api/v1 via Router::nest. Auth routes and /health//config are outside the authorization middleware layer; everything else passes through authorization::authorize which resolves the JWT or API key and injects Extension<MUser>.

Endpoints are split by resource in web/src/endpoints/:

auth.rs          Login, register, OIDC/OAuth2
builds/          Build detail, log streaming, graph, downloads, direct build
caches.rs        Cache CRUD + Nix cache protocol handlers
commits.rs       Commit lookup
evals.rs         Evaluation detail, abort, log streaming
mod.rs           Health, config, 404 handler
projects/            Project CRUD, members, SSH key, cache subscriptions, worker registration
tasks.rs      Task CRUD, entry points, evaluate trigger
user.rs          Profile, API keys, settings
workers.rs       Connected worker list (superuser / global stats)

The Nix binary cache endpoints (/cache/{cache}/…) are registered at the root router, outside /api/v1, to comply with the Nix cache protocol.

Database¶

PostgreSQL is the only supported database. Migrations are in migration/src/ and applied by running cargo run -p migration.

All timestamps are NaiveDateTime (UTC, stored without timezone). The NULL_TIME constant (1970-01-01 00:00:00) is used as a sentinel for "never" (e.g. last_login_at).

Frontend¶

Standalone Angular 22 SPA in frontend/. Communicates exclusively with the backend REST API. Built as static files, served by NGINX in production.

Key patterns: standalone components, Angular signals (signal(), computed()), the in-repo gr-ui component layer on @angular/cdk for UI, Apache ECharts for metric charts, SCSS variables from _variables.scss.

CLI¶

Independent Rust crate in cli/. Uses the connector sub-crate for typed HTTP calls to the REST API. Auth state is stored in ~/.config/gradient/config.toml ($XDG_CONFIG_HOME when set).