Tests¶

This page documents all unit and integration tests in the Rust backend workspace (across 8 crates, including the score scoring crate). Run them with:

cargo test --workspace --tests
# core doctests require a separate invocation (package name shadows stdlib `core`):
cargo test -p core --tests

Tests are grouped by the module under test.

Architecture overview¶

The test infrastructure is built around a set of trait abstractions and fakes that replace production dependencies (nix-daemon, filesystem, WebSocket) with in-memory implementations. The diagram below shows how production and test code relate:

                     ┌─────────────────────────────────────────┐
                     │              proto crate                │
                     │                                         │
                     │  traits.rs                              │
                     │  ┌───────────────────────────────────┐  │
                     │  │  WorkerStore    (has_path)        │  │
                     │  │  DrvReader      (read_drv)        │  │
                     │  │  JobReporter    (report_*)        │  │
                     │  └───────────────────────────────────┘  │
                     │                                         │
                     │  scheduler/                             │
                     │  ┌────────────────┐  ┌───────────────┐  │
                     │  │   JobTracker   │  │   WorkerPool  │  │
                     │  │   (pending/    │  │   (connected  │  │
                     │  │    active)     │  │   workers)    │  │
                     │  └────────────────┘  └───────────────┘  │
                     │         └──────────┬────────────┘       │
                     │                    v                    │
                     │              ┌────────────┐             │
                     │              │ Scheduler  │             │
                     │              └────────────┘             │
                     └─────────────────────────────────────────┘
                                         |
              ┌──────────────────────────┼───────────────────────────┐
              v                          v                           v
   ┌─────────────────────┐   ┌──────────────────────┐   ┌─────────────────────┐
   │    worker crate     │   │  test-support crate  │   │     core crate      │
   │                     │   │                      │   │                     │
   │  LocalNixStore      │   │  FakeWorkerStore     │   │  DerivationResolver │
   │    impl WorkerStore │   │    impl WorkerStore  │   │  (trait)            │
   │                     │   │                      │   │                     │
   │  FsDrvReader        │   │  FakeDrvReader       │   │  parse_drv()        │
   │    impl DrvReader   │   │    impl DrvReader    │   │                     │
   │                     │   │                      │   │                     │
   │  JobUpdater         │   │  RecordingJobReporter│   │  FakeDerivation-    │
   │    impl JobReporter │   │    impl JobReporter  │   │    Resolver         │
   │                     │   │                      │   │                     │
   │  evaluate_          │   │  StoreFixture        │   │  FakeNixStore-      │
   │    derivations_with │   │   (951 real .drv)    │   │    Provider         │
   └─────────────────────┘   └──────────────────────┘   └─────────────────────┘

Test fakes & fixtures¶

All test fakes live in backend/test-support/src/fakes/. Each fake implements a production trait and substitutes the real dependency with an in-memory version.

StoreFixture - real derivation trees¶

StoreFixture (test-support/src/fakes/store_fixture.rs) loads real Nix .drv files from the test/ directory (951 derivations for hello-2.12.3). It performs a BFS from the entry-point derivation through inputDrvs to build the full closure and populates fakes.

  test/
  ├── output          # single line: /nix/store/<hash>-hello-2.12.3.drv
  └── store/          # 951 real ATerm .drv files
      ├── 7mdg60dr...-hello-2.12.3.drv   (entry point)
      ├── abc123...-glibc-2.40.drv
      └── ...

         load_store("test/")
               │
               v
  ┌────────────────────────────────────────┐
  │     StoreFixture                       │
  ├────────────────────────────────────────┤
  │ entry_point: String                    │  ← "/nix/store/7mdg...-hello-2.12.3.drv"
  │ derivations: Vec<...>                  │  ← 951 DiscoveredDerivation structs
  │ tree: HashMap<...>                     │  ← drv_path → [dependency drv_paths]
  │ parsed: HashMap<...>                   │  ← drv_path → Derivation (parsed ATerm)
  │ raw_drvs: HashMap<...>                 │  ← drv_path → raw bytes (for FakeDrvReader)
  │ resolver: FakeDerivationResolver       │
  │ store: FakeNixStoreProvider            │
  └────────────────────────────────────────┘
               │
               v
       mark_all_built()  →  remove_random_subtrees(0.5, seed)
               │
               v
  ┌─────────────────────────────────────────┐
  │ Partial store: some outputs present,    │
  │ some missing. ready_to_build() returns  │
  │ derivations whose deps are all built    │
  │ but which aren't built themselves.      │
  └─────────────────────────────────────────┘

Key methods:

proto - Wire Message Serialization¶

File: backend/proto/src/tests.rs Run: cargo test -p proto

All tests in this module verify the rkyv serialization round-trip: a message is serialized to bytes and deserialized back, then compared with assert_eq!.

The proto protocol runs over a persistent WebSocket (/proto) with rkyv binary framing. The three sub-flows below show which messages each test exercises.

Handshake & Auth Sequence¶

sequenceDiagram
    participant W as Worker (Client)
    participant S as Server

    W->>S: InitConnection<br/>(version, worker_id, capabilities)
    note over W,S: tests: init_connection_roundtrip

    S->>W: AuthChallenge<br/>(peers: Vec<peer_id>)

    W->>S: AuthResponse<br/>(tokens: Vec<(peer_id, token)>)

    alt auth succeeded
        S->>W: InitAck<br/>(version, capabilities,<br/>authorized_peers, failed_peers)
        note over W,S: tests: init_ack_roundtrip
    else auth failed / protocol error
        S->>W: Error<br/>(code, message)
        note over W,S: tests: error_message_roundtrip
    end

Job Dispatch Sequence¶

sequenceDiagram
    participant W as Worker (Client)
    participant S as Server

    W->>S: RequestJobList
    note over W,S: tests: request_job_list_roundtrip

    loop until is_final
        S->>W: JobListChunk<br/>(candidates: [JobCandidate], is_final: false)
        note over W,S: tests: job_list_chunk_roundtrip
    end
    S->>W: JobListChunk<br/>(candidates: [], is_final: true)
    note over W,S: tests: job_list_chunk_final_roundtrip

    S->>W: AssignJob<br/>(FlakeJob { tasks, repo_url, commit,<br/>wildcards, timeouts })
    note over W,S: tests: assign_job_roundtrip

    alt worker accepts
        W->>S: AssignJobResponse<br/>(accepted: true)
        note over W,S: tests: assign_job_response_roundtrip
    else worker rejects
        W->>S: AssignJobResponse<br/>(accepted: false, reason: "…")
        note over W,S: tests: assign_job_response_reject_roundtrip
    end

Drain & Shutdown Sequence¶

sequenceDiagram
    participant W as Worker (Client)
    participant S as Server

    S->>W: Draining
    note over W,S: tests: server_draining_roundtrip
    note over W: Worker finishes in-flight job,<br/>then closes connection

Test Table¶

proto::scheduler::jobs - Job Tracker¶

File: backend/scheduler/src/jobs.rs Run: cargo test -p proto

JobTracker is a dual-queue (pending + active) that tracks jobs through their lifecycle. Tests exercise the state machine directly without any async runtime.

Job lifecycle state machine¶

                     add_pending()
                          │
                          v
                    ┌──────────┐
                    │ PENDING  │◄────────────────────┐
                    └────┬─────┘                     │
                         │                           │
          receive_scores(missing: 0)          release_to_pending()
          or take_empty_required()            or worker_disconnected()
                         │                           │
                         v                           │
                    ┌──────────┐                     │
                    │  ACTIVE  │─────────────────────┘
                    └────┬─────┘
                         │
                   remove_active()
                         │
                         v
                      (done)

Peer-based filtering¶

  ┌─────────────────────────────────────────────────┐
  │               JobTracker                        │
  │                                                 │
  │  Pending:                                       │
  │    "ja" → PendingEvalJob { peer_id: peer_A }    │
  │    "jb" → PendingEvalJob { peer_id: peer_B }    │
  │                                                 │
  │  candidates_for_worker(authorized={peer_A})     │
  │    → returns only "ja"                          │
  │                                                 │
  │  candidates_for_worker(None)                    │
  │    → returns both "ja" and "jb" (open mode)     │
  └─────────────────────────────────────────────────┘

proto::scheduler::worker_pool - Worker Registry¶

File: backend/scheduler/src/worker_pool.rs Run: cargo test -p proto

WorkerPool is an in-memory registry of connected workers with capabilities, authorized peers, and assigned-job tracking.

Worker lifecycle¶

  register()                           unregister()
      │                                     │
      v                                     v
  ┌──────────┐  update_capabilities()  ┌──────────┐
  │CONNECTED │───────────────────────► │CONNECTED │──► returns assigned
  │(default  │  update_authorized()    │(updated) │    job IDs
  │ caps)    │                         └──────────┘
  └──────────┘
       │
       │ mark_draining()
       v
  ┌──────────┐
  │ DRAINING │  (still connected, finishes in-flight jobs)
  └──────────┘

proto::scheduler - Scheduler Coordination¶

File: backend/scheduler/src/scheduler_tests.rs Run: cargo test -p proto

Integration tests for the Scheduler struct, which coordinates WorkerPool and JobTracker behind RwLocks. Uses a mock database (no real Postgres).

Job flow through the scheduler¶

sequenceDiagram
    participant DB as MockDB
    participant S as Scheduler
    participant WP as WorkerPool
    participant JT as JobTracker

    Note over S: enqueue_eval_job("j1", job)
    S->>JT: add_pending("j1", job)

    Note over S: register_worker("w1", caps, peers)
    S->>WP: register("w1", caps, peers)

    Note over S: get_job_candidates("w1")
    S->>WP: authorized_peers_for("w1")
    WP-->>S: {peer_A}
    S->>JT: candidates_for_worker({peer_A})
    JT-->>S: [JobCandidate("j1")]

    Note over S: consider_scores("w1", [{j1, missing:0}])
    S->>JT: receive_scores("w1", [{j1, 0}])
    JT-->>S: Assignment(j1)
    S->>WP: assign_job("w1", "j1")

    Note over S: worker disconnects
    Note over S: unregister_worker("w1")
    S->>WP: unregister("w1") → ["j1"]
    S->>JT: worker_disconnected("w1") → ["j1"]
    Note over JT: j1 back in pending

test_support::fakes::store_fixture - Store Fixture¶

File: backend/test-support/src/fakes/store_fixture.rs Run: cargo test -p test-support

Tests for the StoreFixture itself, validating that real .drv files from test/ are loaded, parsed, and traversed correctly (951 derivations).

Build-wave convergence¶

  Wave 0 (nothing built):
    ready_to_build() → leaf nodes only (no dependencies)
       mark_built(leaves)

  Wave 1:
    ready_to_build() → nodes whose deps are all leaves
       mark_built(wave 1)

  Wave 2:
    ready_to_build() → next frontier
       ...

  Wave N:
    ready_to_build() → [] (everything built)
       assert: built().len() == total derivations

worker::executor::build - Per-Build Metrics¶

File: backend/worker/src/executor/build.rs, backend/worker/src/metrics/cgroup.rs Run: cargo test -p worker executor::build

Tests for best-effort per-build resource capture. The worker records wall-clock build_time_ms for every build. When --build-metrics is enabled, CPU time comes from the daemon's build result (cpu_user + cpu_system, read from the cgroup by nix before it tears the cgroup down), while peak RAM and disk I/O are sampled live from the build's cgroup - located via nix's <state-dir>/cgroups/<uid> map (newest entry written after the build started, since nix destroys the cgroup at build end). Stale map entries and idle/ambiguous cases degrade to None.

worker::metrics::throughput - Passive Throughput EWMA¶

File: backend/worker/src/metrics/throughput.rs Run: cargo test -p worker metrics::throughput

Tests for the thread-safe EWMA accumulators that turn real NAR transfers and per-build disk I/O into the network_speed_mbps / disk_speed_mbps heartbeat values.

worker::executor::eval - Evaluation Closure Walk¶

File: backend/worker/src/executor/eval.rs Run: cargo test -p worker

Tests for evaluate_derivations_with(), which performs a BFS closure walk from entry-point .drv files through inputDrvs to discover all derivations and check their substitution status. Tests use FakeDerivationResolver, FakeDrvReader, FakeWorkerStore, and RecordingJobReporter - no real nix-daemon, filesystem, or WebSocket connection.

Evaluation data flow (test configuration)¶

  FakeDerivationResolver         FakeDrvReader
  ┌───────────────────────┐      ┌───────────────────────┐
  │ list_flake_derivations│      │ from_raw_drvs(        │
  │   "repo" → ["hello"]  │      │   fixture.raw_drvs    │
  │                       │      │ )                     │
  │ resolve_derivation_   │      │                       │
  │   paths               │      │ read_drv(path)        │
  │   "hello" → entry.drv │      │   → raw bytes         │
  └───────────┬───────────┘      └───────────┬───────────┘
              │                              │
              v                              v
  ┌──────────────────────────────────────────────────────┐
  │          evaluate_derivations_with()                 │
  │                                                      │
  │  1. list attrs via resolver                          │
  │  2. resolve attrs → drv paths via resolver           │
  │  3. BFS: read .drv → parse → extract outputs/deps    │
  │  4. Check has_path() for substitution                │
  │  5. Report EvalResult via reporter                   │
  └──────────┬───────────────────────────────┬───────────┘
             │                               │
             v                               v
  FakeWorkerStore                 RecordingJobReporter
  ┌───────────────────────┐       ┌──────────────────────┐
  │ has_path()            │       │ events:              │
  │   → present set       │       │   EvaluatingDervs    │
  │   (from fixture)      │       │   EvalResult {       │
  └───────────────────────┘       │     derivations,     │
                                  │     warnings         │
                                  │   }                  │
                                  └──────────────────────┘

core::types::wildcard - Evaluation Wildcard Parsing¶

File: backend/gradient-types/src/wildcard.rs
Run: cargo test -p core --tests

Tests for the Wildcard type used in project evaluation patterns. Parsing is via FromStr; the inverse is Display. get_eval_str() produces the Nix attribute-set expression passed to the evaluator.

Valid patterns¶

Exclusion restrictions¶

get_eval_str() - Nix expression output¶

Bare special characters¶

Quoted special characters¶

Invalid patterns¶

core::nix::url - Repository & Flake URL Parsing¶

File: backend/gradient-nix/src/url.rs
Run: cargo test -p core --tests

Tests for RepositoryUrl (stored in the database, used for display and git operations) and NixFlakeUrl (passed to nix flake commands, always includes ?rev=).

RepositoryUrl - normalization¶

RepositoryUrl::from_str normalizes certain schemes for Nix compatibility. All others are preserved or rejected.

NixFlakeUrl - flake reference construction¶

NixFlakeUrl::new(url, rev) constructs a ?rev=<sha1> flake URL. The revision must be a full 40-character SHA-1.

core::db::derivation - .drv File Parsing¶

File: backend/gradient-db/src/derivation.rs
Run: cargo test -p core --tests

Tests for parse_drv, which parses the textual Derive(…) format produced by nix derivation show. The fixture derivation used by these tests is:

Derive(
  [("out","/nix/store/abc-hello","","")],
  [("/nix/store/xyz.drv",["out"])],
  ["/nix/store/src"],
  "x86_64-linux",
  "/nix/store/bash",
  ["-e","/nix/store/builder.sh"],
  [("name","hello"),("requiredSystemFeatures","kvm big-parallel"),("system","x86_64-linux")]
)

web::endpoints::badges - CI Badge Rendering¶

File: backend/web/src/endpoints/badges.rs
Run: cargo test -p web

Tests for the SVG badge renderer used by GET /projects/{org}/{project}/badge.

core - Input Validation¶

File: backend/core/tests/input.rs Run: cargo test -p core --tests

Tests for input validation functions used by API endpoints. Each function validates user-supplied values and returns a descriptive error on rejection.

Hex encoding¶

Numeric validation¶

Port validation¶

URL / address parsing¶

Repository URL normalization¶

Index name validation¶

Password policy¶

Wildcard validation (integration layer)¶

core - Source Utilities¶

File: backend/core/tests/sources.rs Run: cargo test -p core --tests

worker::config - Peer Token Parsing¶

File: backend/worker/src/config.rs Run: cargo test -p worker

Tests for WorkerConfig::peer_tokens() and WorkerConfig::resolve_tokens_for_challenge(), the two functions responsible for reading peer-to-token pairs from --peers / --peers-file and expanding them during challenge-response auth.

Token source precedence¶

  peers_file set? ──yes──► read file contents  ──► parse lines
                  │
                  no
                  │
  peers set?    ──yes──► use inline string     ──► parse lines
                  │
                  no
                  │
                  └──────► return [] (open mode, no auth)

Line format¶

  peer_id:token64    →  ("peer_id", "token64")  ✓
  *:token64          →  ("*", "token64")         ✓  (wildcard)
  # comment          →  skipped
  <blank line>       →  skipped
  :token64           →  skipped (empty peer_id)
  peer:              →  skipped (empty token)
  nocolon            →  skipped (no separator)
  peer:short_token   →  skipped (token < 64 chars)

peer_tokens() tests¶

resolve_tokens_for_challenge() tests¶

worker::credentials - Credential Store¶

File: backend/worker/src/credentials.rs Run: cargo test -p worker

Tests for CredentialStore, a shared (cloneable) in-memory store for the two credential kinds the server sends during a job: a UTF-8 signing key and raw SSH private key bytes. Both are wrapped in SecureString/SecureBytes to avoid accidental logging.

  CredentialStore (Arc<Mutex<Inner>>)
  ┌───────────────────────────────────┐
  │ signing_key: Option<SecureString> │  ← UTF-8 Ed25519 secret key
  │ ssh_key:     Option<SecureBytes>  │  ← raw private key bytes
  ├───────────────────────────────────┤
  │ store_signing_key(bytes) → Ok/Err │
  │ store_ssh_key(bytes)              │
  │ signing_key() → Option<&Secure…>  │
  │ ssh_key()     → Option<&Secure…>  │
  │ clear()                           │
  └───────────────────────────────────┘

worker::scorer - Job Candidate Scoring¶

File: backend/worker/src/scorer.rs Run: cargo test -p worker

Tests for score_candidates(), which annotates each JobCandidate with the number of its required_paths that are absent from the worker's local store. The scheduler uses this score to prefer jobs whose inputs are already cached.

  score_candidates(candidates, store)
       │
       ├─ for each candidate:
       │     missing = required_paths not in store.has_path()
       │     → JobScore { job_id, missing }
       │
       └─ return Vec<JobScore>

score - Scheduler Scoring Policies¶

File: backend/score/src/policy.rs, backend/score/src/context.rs, backend/score/src/rules/*.rs Run: cargo test -p score

Tests for the pluggable scheduler scoring crate: each ScoreRule's individual contribution, the lazy ScoringCtx providers, and the composed default / resource-aware policies. See scheduler scoring.

scheduler::history - Build History Prediction¶

File: backend/scheduler/src/history.rs Run: cargo test -p scheduler

Tests for the closure-size-bucketed prediction that feeds ResourceFitRule and DiskAffinityRule: log2-MiB bucketing, neighbour-bucket widening when a bucket is sparse, and aggregation of peak RAM, average CPU time, disk bytes and OOM rate from derivation_metric rows.

core::db::closure - Derivation Closure Helpers¶

File: backend/gradient-db/src/closure.rs Run: cargo test -p core --tests

Tests for the shared closure helpers (transitive_closure_reachable, output_sizes_by_drv, transitive_closure_size, transitive_closure_sizes) extracted from the web closure-graph endpoint and reused by scheduler scoring. transitive_closure_sizes sizes many roots in one batched walk (used by the dispatch backfill).

worker::metrics - Host Metrics & CPU-Core Score¶

File: backend/worker/src/metrics/mod.rs Run: cargo test -p worker metrics

Tests for the host static/dynamic metrics the worker advertises, including the single-core speed benchmark used for cpu_core_score.

worker::handshake - Challenge-Response Handshake¶

File: backend/worker/src/handshake.rs Run: cargo test -p worker

Tests for perform_handshake(), which drives the full challenge-response auth sequence over a live ProtoConnection. All tests use MockProtoServer - no real Gradient server needed.

Handshake sequence¶

sequenceDiagram
    participant W as Worker (perform_handshake)
    participant S as MockProtoServer

    W->>S: InitConnection(version, worker_id, capabilities)
    S->>W: AuthChallenge(peers: ["peer-a"])
    W->>S: AuthResponse(tokens: [("peer-a", "tok64")])
    S->>W: InitAck(authorized_peers: ["peer-a"], failed_peers: [])
    Note over W: HandshakeResult { authorized_peers, server_version }

worker::job - Job Updater Protocol¶

File: backend/worker/src/job.rs Run: cargo test -p worker

Tests for JobUpdater, which wraps a ProtoConnection and provides typed methods for reporting job progress. Each test uses MockProtoServer and a server_then_client! macro that spawns the server accept task before the client opens its connection (required to avoid deadlock on the single-thread tokio runtime).

  JobUpdater::report_fetching()
       │
       └─► ProtoConnection::send(ClientMessage::JobUpdate {
               job_id: "…",
               update: JobUpdateKind::Fetching,
           })
               │
               └─► MockServerConn::recv() → assert message shape

worker::nar - NAR Push & Upload¶

File: backend/worker/src/nar.rs Run: cargo test -p worker

Tests for push_direct() (chunk-streaming to server) and upload_presigned() (streaming to a presigned URL). Both use MockProtoServer and temp directories; push_direct also validates zstd compression integrity.

push_direct flow¶

  push_direct(job_id, store_path, conn)
       │
       ├─ NarByteStream::new(store_path) → async byte stream
       ├─ zstd-compress chunks (64 KiB each)
       │
       ├─► NarPush { job_id, store_path, offset: 0, data: <bytes>, is_final: false }
       ├─► NarPush { job_id, store_path, offset: N, data: <bytes>, is_final: false }
       └─► NarPush { job_id, store_path, offset: M, data: [],      is_final: true  }

upload_presigned flow¶

  upload_presigned(job_id, store_path, url, conn)
       │
       ├─ stream NAR → HTTP PUT to presigned URL
       ├─ compute sha256 of raw NAR bytes
       │
       └─► NarReady { job_id, store_path, sha256: "sha256:<hex>", nar_size: N }

worker::executor::sign - Nix Path Signing Helpers¶

File: backend/worker/src/executor/sign.rs Run: cargo test -p worker

Tests for the two private pure functions that convert between hash formats required by fingerprint_path. No nix-daemon, filesystem, or network access.

Hash conversion pipeline¶

  sign_one_path()
       │
       ├─ query_path_info() → path_info.nar_hash  (SRI format: "sha256-<base64>")
       │
       ├─ sri_to_nix_hash("sha256-<base64>")
       │       │
       │       ├─ strip "sha256-" prefix
       │       ├─ base64-decode → raw bytes
       │       └─ nix_base32_encode(raw) → "sha256:<nix-base32>"
       │
       └─ fingerprint_path(store_dir, store_path, nar_hash_nix, nar_size, refs)

worker::executor::fetch - Flake Fetching¶

File: backend/worker/src/executor/fetch.rs Run: cargo test -p worker

Tests for fetch_repository(), which clones or fetches a git repository for a given commit and optionally injects an SSH key from CredentialStore. Tests use RecordingJobReporter - no real git server or WebSocket needed.

core::types::secret - Secret Memory Wrappers¶

core::state_machine::build - Build Status Machine¶

core::state_machine::eval - Evaluation Status Machine¶

core::forge::github_app - GitHub App JWT & Webhook Verification¶

core::ci::webhook - Webhook Signing & Encryption¶

core::ci::trigger - Evaluation Trigger Guard¶

core::state::provisioning - Inbound Integration Lookup¶

core::sources::ssh_key - SSH Key Formatting¶

web::endpoints::caches::nar - NAR URL Resolver¶

Unit tests for resolve_effective_hash_db, which maps the file-hash embedded in a narinfo URL back to the store-hash key used by the NAR blob storage.

core::sources::cache_key - Cache Signing Key Lifecycle¶

builder::status - Evaluation Terminal Status¶

worker::wildcard_walk - Flake Output Wildcard Traversal¶

Pure traversal driving the cursor-backed flake discovery (replaces the retired eval.nix); unit-tested against a stub attr tree.

worker::worker_pool::resolver - Attr Path Matching¶

The crash-isolation policy (resolve_chunk) is driven through an injected resolve_once stub so the bisection state machine is exercised without real eval subprocesses. RSS recycling is parent-side via EvalWorker::rss_bytes (reads /proc/<pid>/statm).

worker::executor::build - BasicDerivation Construction¶

worker::worker - Persistent Worker ID¶

worker::eval_worker - Eval Worker Protocol Serde¶

proto::scheduler::dispatch_tests - Dispatch Loop Functions¶

File: backend/scheduler/src/dispatch_tests.rs Run: cargo test -p proto

Integration tests for the two background dispatch functions that feed the in-memory scheduler from the database. Both are normally private but are exposed as pub(crate) for testability.

Tests use a staged MockDatabase and a real Scheduler instance; after each call the pending job count is asserted directly on the scheduler.

dispatch_queued_evals - DB call sequence¶

  dispatch_queued_evals(scheduler)
       │
       ├─ 1. EEvaluation::find().filter(status=Queued).all()       Q
       │
       └─ for each eval (if not already in scheduler):
               ├─ 2. ECommit::find_by_id(commit_id).one()          Q
               │
               └─ 3. organization_id_for_eval:
                       ├─ if eval.project = Some(pid):
                       │     EProject::find_by_id(pid).one()       Q
                       └─ else (direct build):
                             EDirectBuild::find()
                               .filter(evaluation=id).one()        Q

dispatch_ready_builds - DB call sequence¶

  dispatch_ready_builds(scheduler)
       │
       ├─ 1. EBuild::find().from_raw_sql(ready_builds_query).all() Q
       │       (Queued builds with no unsatisfied dependencies)
       │
       └─ for each build (if not already in scheduler):
               ├─ 2. EDerivation::find_by_id(drv_id).one()         Q
               ├─ 3. EEvaluation::find_by_id(eval_id).one()        Q
               └─ 4. organization_id_for_eval (see above)          Q

Group F: dispatch_queued_evals¶

Group F: dispatch_ready_builds¶

proto::scheduler::handler_tests - Scheduler DB Handler Functions¶

File: backend/scheduler/src/handler_tests.rs Run: cargo test -p proto

Integration tests for the six handler functions that the scheduler calls when workers report results. These are the most critical code paths in the system: they insert derivations and builds into the database, cascade failures through dependency edges, and determine when an evaluation is complete.

Tests drive the handler functions directly (eval::handle_eval_result, build::handle_build_job_completed, etc.) using SeaORM's MockDatabase with staged query/exec results - no real Postgres required.

MockDatabase staging rules¶

SeaORM 1.x on Postgres has a subtle split between two result queues:

  append_query_results  →  consumed by SELECT, UPDATE…RETURNING, find_by_id,
                           and insert_many().exec() (Postgres uses RETURNING)

  append_exec_results   →  consumed by update_many().exec() and single-row
                           insert() with an explicit primary key

The critical non-obvious rule: EEntity::insert_many().exec() on the Postgres backend calls db.query_all() internally (because primary_key = None + support_returning = true), so it consumes from query_results, not exec_results. Each staged query result for an insert_many only needs a row with a valid id: Uuid.

Data flow through handle_eval_result¶

  Worker sends EvalResult
       │
       v
  handle_eval_result(state, job, derivations, warnings)
       │
       ├─ 1. EEvaluation::find_by_id()          Q
       │       abort if status == Aborted
       │
       ├─ 2. EDerivation::find()...all()         Q   (existing drv check)
       │
       ├─ 3. EDerivation::insert_many().exec()   Q   (new drvs only)
       ├─ 4. EDerivationOutput::insert_many()    Q
       ├─ 5. EDerivationDependency::insert_many()Q   (if deps present)
       ├─ 6. EBuild::insert_many().exec()        Q
       │
       ├─ 7. record_evaluation_message() ×N      E   (per warning)
       │
       ├─ 8. EBuild::find() Created builds       Q
       │
       ├─ 9. update_build_status() ×N            Q   (Created → Queued)
       │
       └─ 10. update_evaluation_status()         E + Q

Cascade flow through handle_build_job_failed¶

  handle_build_job_failed(state, build_id, error)
       │
       ├─ EBuild::find_by_id()                              Q
       ├─ update_build_status(build, Failed)                Q  (Building→Failed)
       │
       └─ cascade_dependency_failed(eval_id, failed_drv_id)
               │
               ├─ EBuild::find() Created/Queued builds      Q
               └─ for each candidate build:
                       ├─ EDerivationDependency::find()     Q  (dep edge check)
                       └─ if edge: update_build_status      Q  (→ DependencyFailed)

Group A: handle_eval_result¶

Group B: handle_build_job_completed + check_evaluation_done¶

Group C: handle_build_job_failed + cascade_dependency_failed¶

Group D: handle_build_output¶

Group E: handle_eval_job_completed / handle_eval_job_failed¶

Group G: abort_evaluation¶

Tests for gradient_ci::abort_evaluation, which cascades Aborted to all active builds before aborting the evaluation itself.

DB call sequence:

  abort_evaluation(state, evaluation)
       │
       ├─ guard: if eval.status == Completed → return (no DB calls)
       │
       ├─ 1. EBuild::find()
       │       .filter(evaluation=id)
       │       .filter(status IN Created/Queued/Building)
       │       .all()                                                Q
       │
       ├─ for each active build:
       │     update_build_status(build, Aborted)                    Q  (UPDATE…RETURNING)
       │     spawns fire_build_webhook(Aborted) → returns early
       │     spawns log_finalize (NoopLogStorage → no-op)
       │
       └─ update_evaluation_status(eval, Aborted)
               update_many().exec()                                  E
               find_by_id(eval)                                      Q
               spawns fire_evaluation_webhook(project=None → early return)

Group H: Handler behavioral gaps¶

Tests for error paths and edge cases not covered by Groups A–E.

Group I: Webhook delivery¶

Tests that use eval.project: Some(project_id) to exercise the full webhook delivery path. A real temporary key file is created so decrypt_webhook_secret can succeed. After the handler returns, tokio::task::yield_now().await lets the spawned webhook tasks run on the current_thread runtime. Deliveries are captured by RecordingWebhookClient (returned alongside the state by test_state_recorded).

  handle_build_job_completed()
         │
         ├─ update_build_status(Completed)
         │     spawns ──► fire_build_webhook(Completed)
         │                    ├─ get_build_org_id → eval → project → org
         │                    ├─ find_by_id(derivation) [best-effort]
         │                    └─ fire_webhooks → decrypt → sign → deliver
         │                                                  ↓
         │                                       RecordingWebhookClient.calls()
         │
         └─ update_evaluation_status(Completed)
               spawns ──► fire_evaluation_webhook(Completed)
                              ├─ eval.project = Some → find project → org
                              └─ fire_webhooks → subscription check → deliver

core::storage::nar_extract - NAR Single-File Extractor¶

NAR extraction (core/tests/nar_extract.rs)¶

Tests for extract_file_from_nar_bytes, which decompresses a zstd-compressed NAR from nar_storage and returns the contents of a single file identified by its relative path.

Note: tests use sync #[test] + manual tokio::runtime::Builder::block_on because the core package name shadows stdlib ::core, which breaks #[tokio::test] macro expansion. See the MEMORY.md entry on the "core crate name shadowing" gotcha.

Cache NAR management (issue #260)¶

Backend (backend/web/tests/)¶

• cache_nar_list::list_empty_cache_returns_empty_items
• cache_nar_list::list_private_cache_requires_auth
• cache_nar_list::list_accepts_pagination_query_params
• cache_nar_show::show_returns_full_detail
• cache_nar_show::show_404_when_signature_missing
• cache_nar_stats::stats_returns_aggregates
• cache_nar_available::available_returns_true_when_signature_present
• cache_nar_available::available_returns_false_when_no_cached_path
• cache_nar_delete::delete_unauthenticated_returns_403
• cache_nar_delete::delete_owner_removes_signature_and_writes_audit_row (ignored - requires real-DB harness)

Notes¶

Tests that depend on read-after-write persistence (signature removal, is_cached flip, blob GC) are gated #[ignore] because the existing test harness uses SeaORM MockDatabase. Verifying these behaviors needs a real Postgres backend, which is a follow-up infrastructure task.

Evaluation metrics¶

Unit tests for the per-evaluation metrics pipeline (eval-worker capture -> scheduler persistence + RAM routing -> board read endpoints -> frontend panel):

• Eval-stats delta aggregator (gradient-worker worker_pool::eval_stats) - sums_totals_across_requests, buckets_per_entry_point, and heap_peak_is_max_gauge_not_sum cover diffing cumulative counters per request, bucketing costs per entry-point, and treating the GC heap as a max gauge rather than a sum.
• Per-project eval-RAM window (gradient-scheduler instance::eval_history) - eval_history_maps_row_into_prediction maps the per-project p95 peak-RSS window row into a HistoryPrediction.
• Eval RAM-overshoot scoring (gradient-score rules::resource) - eval_ram_overshoot_routes_to_big_ram_worker asserts ResourceFitRule penalises an RAM-heavy eval on a small worker so it routes to a big-RAM one.
• Board read endpoints (gradient-web endpoints::board) - eval_metric_expr_known_keys_have_units / eval_metric_expr_unknown_is_none cover the closed metric allow-list (so metric can never inject SQL); flake_output_node_maps_to_graph_node maps a flake_output_node::Model to the FlakeGraphNode DTO field-for-field. The org-scoping SQL of get_expensive_evals_by_resource and the EvalAccessContext auth path of get_eval_flake_graph have no local DB harness and are covered by E2E CI.
• Frontend board.service (board.service.spec.ts) - getExpensiveEvalsByResource() and getEvalFlakeGraph() GET their endpoints and unwrap the response array.

Test count summary¶

Integration Tests¶

The NixOS VM integration tests live in nix/tests/gradient/ and are documented in Contributing. They test the full server + worker stack end-to-end, including the proto handshake, job dispatch, cache serving, and declarative state management.

Worker Authentication¶

Both the building and cache integration tests validate the challenge-response authentication flow that workers use to connect to the server:

gradient-building - API-based worker registration: 1. Worker boots and generates a persistent UUID 2. User creates an organization via CLI 3. Worker's UUID is registered with the org via POST /api/v1/orgs/{org}/workers 4. Negative sub-test: Worker restarts WITHOUT a peers file → server challenges, worker has no tokens → Reject(401). Test asserts rejection in journalctl. 5. Positive sub-test: Peers file is written with {peer_id}:{token}, worker restarts with GRADIENT_WORKER_PEERS_FILE → handshake succeeds. Test asserts "handshake successful" in journalctl. 6. Project is created and evaluation + build complete with the authenticated worker.

gradient-cache - Declarative state-managed worker registration: 1. Worker UUID is pre-seeded via systemd.tmpfiles.rules to a known value 2. Server state config registers the worker with a shared token (hashed and stored) 3. Worker's peersFile contains *:{token} (wildcard matches any org UUID) 4. On boot, worker authenticates via challenge-response with the pre-shared token 5. Test asserts "handshake successful" in journalctl before proceeding to builds

gradient-api - Worker registration CRUD: - Tests POST /api/v1/orgs/{org}/workers returns a valid 64-character base64 token - Tests GET /api/v1/orgs/{org}/workers lists the registered worker - Tests DELETE /api/v1/orgs/{org}/workers/{id} removes the registration

gradient-eval - Cursor-driven flake discovery/resolution (single node, drives the --eval-worker subprocess directly over its line-delimited JSON protocol against a committed fixture flake): - Trailing * recovers one level (packages.x86_64-linux.* finds hello, cowsay, and nested.inner); # is non-recursive (only hello, cowsay); an exact-path ! exclusion drops cowsay. - resolve returns hello's .drv; a throw-ing boom attribute isolates as a per-item error without aborting the batch (#139). - The lock-only fingerprint op returns the flake's eval-cache key, and the on-disk cache is named exactly eval-cache-v6/<fingerprint>.sqlite - the path-agreement the fleet eval-cache (#386 L3) relies on to stage/pull blobs.