[MLPerf 6.1] Adding DLRMv4 MLPerf Logging Compliance Checker

[chriscai-amd]

Summary

Registers dlrmv4 as a Training 6.1 benchmark: closed and open rulesets, reference convergence points, log constants, and the result summarizer row. dlrmv4 is the HSTU recommendation benchmark on Yambda-5B, taking the recommendation slot that dlrm_dcnv2 occupied through 6.0.

The three commits split cleanly if you'd prefer them reviewed separately: the closed ruleset and RCPs, then the constants and open ruleset, then the eval-start rule reworked to follow deepseekv3_671b.

Changes

File	Change
benchmark_meta.py	'dlrmv4': 10 in training result counts; 'dlrmv4' in the '6.1' allowed list
rcp_checker/rcp_checker.py	'dlrmv4': 10 in submission_runs; dlrmv4 added to the samples_count allow-set in read_submission_file
rcp_checker/training_6.1.0/rcps_dlrmv4.json	New: three records at global batch 8192 / 16384 / 32768
compliance_checker/training_6.1.0/closed_common.yaml	dlrmv4 appended to the submission_benchmark allow-list
compliance_checker/training_6.1.0/open_common.yaml	Same
compliance_checker/training_6.1.0/closed_dlrmv4.yaml	New closed ruleset
compliance_checker/training_6.1.0/open_dlrmv4.yaml	New open ruleset
compliance_checker/README.md	Lists both new yamls
mllog/constants.py	DLRMV4 plus four log keys
result_summarizer/config.yaml	dlrmv4: ["Benchmark results (minutes)", "Recommendation", "Yambda-5B", "DLRMv4 (HSTU)"]

Convergence is keyed on samples_count

dlrmv4 converges at roughly 0.027 to 0.056 of an epoch. Rounding that to three decimals, as the epoch_num branch does, would collapse distinct seeds onto identical values, so dlrmv4 joins llama31_8b, flux1, qwen35_397b_grpo and the others in reading metadata['samples_count'].

New log keys

Four keys have no existing constant. Three describe parts of the model no current benchmark has:

• opt_sparse_name, opt_sparse_base_learning_rate — the embedding-table optimizer, separate from the dense one.
• opt_learning_rate_warmup_start_lr — the absolute LR the ramp begins from, distinct from the existing warmup step count.

The fourth, eval_every_data_pct, describes the eval grid, which here is a correctness concern rather than bookkeeping, since samples-to-converge is quantized to it.

The remaining fixed hyperparameters map onto existing constants (OPT_ADAM_BETA_1/2, OPT_ADAM_EPSILON, OPT_WEIGHT_DECAY, OPT_GRADIENT_CLIP_NORM, OPT_LR_WARMUP_STEPS, MAX_SEQUENCE_LENGTH).

What the closed rules pin

Only the target learning rate and the global batch size may be tuned. Everything else is fixed: optimizer names (Adam dense, RowWiseAdagrad sparse), Adam betas 0.95 / 0.999, epsilon 1e-8, weight decay 0, gradient clip norm 1.0, warmup of 24000 steps starting from LR 0, max sequence length 4096, and train_samples / eval_samples.

Two constraints are computed from the logged global batch size rather than hardcoded, because a single pinned value would be wrong at two of the three reference batch sizes:

• eval_interval_samples = max(1, round(0.001 * (train_samples // GBS))) * GBS. Convergence is reported on this grid, so samples-to-converge is quantized to it and is only comparable against the RCPs when the grid matches. Every eval_accuracy must land on it.
• Where measurement starts. The reference suppresses early eval passes until floor((4480 * GBS + 135331840) / 3) samples, a batch-size-dependent threshold that is easy to leave at the smallest batch size's default, silently shifting where measurement begins. Following the deepseekv3_671b pattern, this is enforced on the log rather than on a self-reported config value: a FIRST_CHECK on eval_accuracy requires the run's first eval to land on the grid point at or after that threshold — block 25, 31 and 41 at global batch 8192, 16384 and 32768. An earlier revision of this PR compared the logged skip_eval_epoch_pct against the formula instead, which a run could satisfy while still evaluating somewhere else; that key and its constant are now dropped.

The open ruleset keeps the task-defining constants and the AUC 0.75 target and drops the hyperparameter value checks, following the open/closed split used by qwen35_397b_grpo and gpt_oss_20b.

RCP provenance

A 60-run sweep, 20 seeds at each of three batch sizes, on MI350X in BF16.

BS	LR	earliest	mean	latest
8192	1e-6	61.9M	69.3M	80.3M
16384	2e-6	78.0M	87.7M	100.9M
32768	4e-6	98.6M	113.0M	128.5M

Each entry is the first eval reaching AUC 0.75, which is exactly what a compliant run reports, since the reference stops at the first crossing and emits run_stop with status="success".

Note for reviewers: the reference sweep deliberately ran full convergence curves without early stopping, which is the normal way to generate RCPs, so those logs carry no run_stop and their eval grid starts at block 1 rather than at the skip threshold. The extraction rule above is stated explicitly so every number is reproducible from them.

Validation

• Production-shaped compliant logs at all three batch sizes — built from real sweep seeds, filtered to the eval window a submission would produce (first eval at block 25 / 31 / 41, truncated at the first 0.75 crossing, run_stop appended) — pass the compliance checker with --werror, in both closed and open division.
• Six mutations are each rejected with the specific failing check: the wrong batch size's start point (a GBS-16384 run starting at block 25) and a start one block late both fail the new FIRST_CHECK; an eval one sample off the grid fails the divisibility check; and wrong Adam beta, a doubled eval cadence, and a sparse LR diverging from the dense one fail their respective checks.
• The eval-grid formula reproduces all 4,779 eval events across all 60 reference logs with zero off-grid, so the rule cannot reject a valid run.
• All edited YAML parses and all edited Python compiles; no duplicate constant names or key strings introduced.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅