Performance Accelerators

This chapter documents the performance behaviours of Novomodelo’s SDDP solver that a user configures or observes: what to tune and what to read back from the output. Each accelerator addresses a specific cost driver in the training loop and is active by default unless noted otherwise. Understanding them helps users interpret timing statistics, configure cut management strategies, and diagnose performance regressions. The crate-internal data structures and allocation discipline that back these behaviours are documented in the novomodelo developer surface — see See Also.

Diagnosing Performance

The timing tables and the printed Time split hold rank 0’s view, so another rank’s imbalance appears as rank 0’s time in cut_sync_ms, state_exchange_ms and mpi_allreduce_ms. Reading the Time split breakdown explains how to read the printed block. An enumerated backward pass records zero in cut_sync_ms, state_exchange_ms and bwd_load_imbalance_ms.

Symptom	Metric	Knob
Backward time grows each iteration	`time_backward_ms` and `cuts_active` in `training/convergence.parquet`; `cuts_active_after` and `active_after_budget` in `training/cut_selection/iterations.parquet`	`training.cut_selection.selection` and `training.cut_selection.max_active_per_stage`
Lazy-selection solves load many rows	`mean_rows_in_lp` in `training/convergence.parquet` (0 when no lazy selection ran); `lazy_scoring_ms` in `training/timing/iterations.parquet`	The `dynamic` method’s parameters under `training.cut_selection.selection`
Workers wait inside a phase	`fwd_load_imbalance_ms` and `bwd_load_imbalance_ms` in `training/timing/iterations.parquet`; the wait share of each phase in the printed time split	Backward: `by_node` with `block_size` under `training.parallelism.backward_scheduler`, see By-Node Scheduling (no effect while lazy selection is active). Forward: a `training.selection.forward_passes` that is a multiple of the worker count (ranks times `--threads`)
Workers idle for want of trajectories	`forward_passes` in `training/convergence.parquet` against `distribution.world_size` and `distribution.threads_per_rank` in `training/metadata.json`	Raise `training.selection.forward_passes`, or run fewer ranks or fewer threads per rank; see Parallel Execution Model
Collective time on multi-rank runs	`cut_sync_ms`, `state_exchange_ms` and `mpi_allreduce_ms` in `training/timing/iterations.parquet`	Balance the work as in the two rows above, or run fewer ranks with more `--threads` each, typically one rank per node; see Parallel Execution Model
Lower-bound evaluation takes a growing share of iteration time	`lower_bound_ms` in `training/timing/iterations.parquet`	None: the evaluation is serial on rank 0, one LP solve per stage-0 opening; see Lower Bound Evaluation
Solves retry	`lp_retries` and `retry_attempts` in `training/solver/iterations.parquet`; `retry_level` in `training/solver/retry_histogram.parquet`; the LP prescaling diagnostics in `training/scaling_report.json`	The per-phase solver profiles `training.solver.backward`, `training.solver.forward` and `simulation.solver`; `modeling.cost_scale_factor`
Slow start	`setup.load_seconds`, `setup.stochastic_fit_seconds`, `setup.production_fit_seconds`, `setup.evaporation_fit_seconds` and `setup.broadcast_seconds` in `training/metadata.json` (collected on rank 0)	For the stochastic fit, export the fitted artifacts once and reuse them: `exports.stochastic` and Exporting Stochastic Artifacts

Solver Safeguards

When HiGHS returns a non-terminal error, the solver automatically escalates through a bounded retry sequence rather than surfacing the failure. The caller never sees intermediate failures — only the final solution or a single classified solver error. Escalation is capped by a built-in per-attempt iteration limit and per-level and overall wall-clock budgets, so a pathological solve cannot stall the run indefinitely.

How far escalation went on a given solve is recorded in the retry_level output column (values 0–11) and written to training/solver/retry_histogram.parquet for post-run analysis; a solve that never retries adds no histogram row; level 0 is the first retry level. The escalation strategy itself — which options each level applies — is a solver-internal detail documented in the novomodelo-solver README.

LP Scaling

Before each stage’s LP template is built, Novomodelo applies its own geometric-mean row/column prescaler that normalizes the constraint-matrix coefficients toward 1.0, improving numerical conditioning. The prescaling mechanism and its exact-unscaling guarantee are documented in the math layer under LP Layout and Scaling.

Cost Scale Factor

The objective is divided by a cost-scale factor (default 1_000_000.0, configurable as modeling.cost_scale_factor) to reduce the magnitude of objective coefficients, improving simplex numerical stability.

Because the prescaler already conditions the matrix, HiGHS’s internal scaling (simplex_scale_strategy) is disabled by default in every solver profile. It is overridable per phase via the solver profile’s scale field: selecting solver_scaling re-enables solver-side scaling and forgoes the exact single-unscaling guarantee. See Solver profile fields for the field and its trade-off.

The scaling diagnostics are written to training/scaling_report.json after template construction, documenting the coefficient range before and after scaling for each stage.

Cut Management Pipeline

As training progresses, the row pool grows and LP solve times increase. Novomodelo provides a two-stage row management pipeline (Stage 1, the selection pass, then Stage 2, the budget pass) to control this growth while preserving convergence guarantees.

The pipeline runs after each iteration’s backward pass and cut synchronization:

Stage 1: Strategy-Based Selection

Four strategies are available, configured through training.cut_selection in config.json; the Configuration reference owns the keys, tolerances and defaults:

Strategy	Selection rule
`level1`	Keeps every cut within the tolerance of the per-state maximum at some visited state and deactivates the other eligible cuts
`lml1`	Keeps, at each visited state, only the oldest cut within the tolerance of the maximum
`domination`	Applies the `level1` test with its own tolerance
`dynamic`	Lazy incremental scheme: loads a resident subset of the pool at each solve and grows it with the cuts the current LP solution violates; deactivates nothing

level1, lml1, and domination respect check_frequency: the selection pass runs only at iterations that are multiples of check_frequency. The first stage and the terminal pool are always exempt from it (the first stage’s rows drive the lower bound and are never backward-pass successors; the backward pass adds no cut to the terminal pool).

These three strategies evaluate every populated cut at every visited forward-pass state, including cuts currently flagged inactive, so a deactivated cut can be reactivated when it later achieves the maximum at some state. Their semantics are in Periodic-Pruning Strategies.

dynamic (Dynamic Cut Selection, DCS) operates differently: it is a per-solve lazy selection loop that adds cuts on demand rather than deactivating from a full pool scan. It does not respect check_frequency. See Dynamic Cut Selection for the procedure.

Stage 2: Budget Enforcement

A hard-cap safety net on LP size, enabled via max_active_per_stage. When the number of active rows exceeds the budget after the selection pass, the pool evicts rows sorted by staleness (last_active_iter ascending, then active_count ascending). The cap applies to every pool, including a terminal pool that holds a loaded boundary policy’s cuts; only rows generated in the current iteration are exempt, so a cap below the number of active imported boundary cuts evicts some of them.

Unlike the selection pass, budget enforcement runs every iteration (not gated by check_frequency).

Why it matters: High-parallelism configurations (many forward passes, few iterations) accumulate more active rows than low-parallelism configurations (fewer forward passes, more iterations), making each backward LP solve proportionally more expensive. Bounding LP size makes high-parallelism configurations viable without unbounded solve-time growth.

Observability

The row management pipeline writes per-stage statistics to training/cut_selection/iterations.parquet, where each column is defined. The knob table names the columns that diagnose backward-time growth.

Parallel Execution Model

A run has two levels of parallelism: MPI ranks, and worker threads inside each rank. --threads sets the number of worker threads per rank (default 1, at least 1); the launcher sets the rank count (see HPC & Cluster Deployment).

The Execution block of the run banner reports both levels. The excerpt below shows an example two-rank run with two threads per rank on one host; it omits the Solver line and writes the hostname as <host>:

Execution
  Backend:   MPI (MPICH 4.3.2, MPI 4.1)
  Threads:   Funneled, 2 rayon threads per rank
  Layout:    2 ranks on <host>

The Threads: line gives the MPI thread level and the worker threads per rank; the Layout: line gives the rank count and the host. A run on several hosts shows across N nodes there, with the ranks of each host listed below it.

Work Partition

Under a sampled forward pass, the forward_passes trajectories of an iteration (see training.selection) are split over the ranks in contiguous blocks. Each rank takes forward_passes / ranks of them, rounded down, and the first forward_passes mod ranks ranks take one extra: 50 passes on 4 ranks give 13, 13, 12 and 12, and ranks beyond the number of passes receive none. A rank then splits its block statically over its worker threads the same way, so the assignment of trajectories to threads depends on the counts only, never on thread scheduling. Simulation scenarios are split over ranks and threads the same way.

Each rank runs the backward pass for the trial points of its own forward passes; the work units handed to its worker threads are defined under Backward-Pass Scheduling. Under an enumerated forward pass, the backward pass instead splits the successor outcomes of each node across the ranks, which allgather them so that every rank derives the same cut.

Every rank must run with the same --threads value: the backward pass of a sampled forward pass compares the worker counts across the ranks and stops the run when they differ.

Backward-Pass Scheduling

By default (by_scenario), each parallel backward-pass work unit is one whole trial point. The optional by_node scheduler uses finer (trial point, opening block) work units for better load balance when trial points solve in variable time. Either way the produced cut set and lower bound are independent of worker and thread count. See parallelism.backward_scheduler for the configuration, and the novomodelo-sddp README for the scheduler and inter-rank communication internals.

By-Node Scheduling

The by_node scheduler shrinks the work unit from one trial point to one (trial point, opening block) pair, giving the work-stealing counter more, finer-grained units to hand out. This improves load balance specifically when trial points solve in variable time: a worker that finishes a cheap trial point quickly can pick up a block from a still-running expensive one instead of idling until an entire trial point frees up.

Within a stage, opening-blocks are claimed hardest-first: blocks are ranked by the previous iteration’s mean simplex pivot cost per (stage, block), descending, so the most expensive blocks are claimed early rather than left for whichever worker happens to reach them last. This claim order changes only which worker processes which block, and when — it is result-neutral: the produced cut set and the training lower bound are identical to canonical (whole-trial-point) claiming.

See parallelism.backward_scheduler for the config field that enables by_node and sets its block_size.

Per-Phase Solver Profiles

Each algorithmic phase — forward sweep, backward sweep, and simulation — can override the built-in solver profile (feasibility tolerances, the simplex update limit, pricing strategy, the scale field, and more) via training.solver.backward, training.solver.forward, and simulation.solver. See Solver profile fields for the full field list, the built-in profile each phase starts from, and the single-unscaling trade-off carried by scale.

Overriding the backward phase with tighter feasibility tolerances can reduce backward-pass solve-time variance, which improves load balance across worker threads and shortens wall-clock training time.

Forward Pass and Simulation

Under a sampled forward pass and in simulation, scenarios are assigned to workers by the static partition described under Work Partition. Within each scenario the stage LP is loaded once. Per stage, the forward-pass and simulation solves patch the incoming-state column bounds, the stochastic NCS generation column bounds, and the load-balance and inflow row bounds.

Lower Bound Evaluation

The lower-bound evaluation (solving a stage-0 LP for every opening in the tree) runs as a single-threaded serial loop on rank 0. Each opening patches correctness-critical per-opening state (e.g. NCS column bounds) on a shared solver in sequence, so the step is not parallelized.

Communication and Threads

The ranks exchange data through blocking MPI collectives, so communication does not overlap computation. MPI is initialized at the Funneled thread level, which the Threads: line of the Execution block names: only one thread of a rank makes MPI calls. An allgatherv gives every rank the data of all ranks, an allreduce combines a value over all ranks, and a broadcast sends a value from rank 0 to the others. On every run the main collectives are:

an allgatherv of the forward-pass costs, reduced in canonical order, so the result does not depend on the rank count;
allreduce sums of integer counters, among them the solve counts of the printed summary;
a broadcast of the lower bound, which rank 0 evaluates (see Lower Bound Evaluation) and the other ranks wait for.

Under a sampled forward pass the backward pass adds:

an allgatherv of the trial states of all ranks at the start of each backward stage (timed as state_exchange_ms);
a per-stage allgatherv of the cuts generated in the backward pass, at the end of each backward stage, which every rank waits on (timed as cut_sync_ms);
an allreduce sum of the per-iteration cut-binding increments.

The enumerated backward pass has neither the state exchange nor the cut exchange; it uses an allgatherv of the successor outcomes of each node instead (see Work Partition). The communicator backends are documented in the novomodelo-comm README.

Each LP solve runs on one thread: the bundled HiGHS is built without its default threads, so during the forward pass, the sampled backward pass and simulation the number of LPs a rank solves at the same time is its --threads. The enumerated backward pass runs on one worker thread per rank.

Rank 0 writes the training outputs. The timing tables and the printed Time split show rank 0’s view: the phase wall, wait, and serial values are rank 0’s own, not an aggregate over the ranks. The printed solve counts are sums over all ranks, and the printed solve time is the mean per worker over all ranks.

Memory and Case Access

Each rank holds its own full copy of the case data, the opening tree, the cut pool, and the stage templates; ranks on one node do not share memory. The cut pool is sized at startup from the iteration budget (the largest iteration_limit in the stopping rules) times the forward passes, for every stage. The memory of every rank is set by that size, and placing more ranks on a node multiplies that node’s memory use.

Rank 0 loads the case and broadcasts the system data, the configuration, and any user-supplied opening tree; the other ranks rebuild the stochastic context and the hydro models from the case directory. Every rank must read the case directory, so it must be visible at the same path on every node (a shared filesystem or a staged copy). A resume or warm start also loads the policy checkpoint on every rank, from the output directory.