Stop the curriculum schedule starting below min_difficulty

[vineethsaivs]

Symptom

The curriculum schedule can start below the min_difficulty it is configured with, and for one pair the tutorial itself recommends it starts at 0.

from deepspeed.runtime.data_pipeline.curriculum_scheduler import CurriculumScheduler

def sched(min_d, step):
    return CurriculumScheduler({
        "min_difficulty": min_d, "max_difficulty": 1024, "schedule_type": "fixed_linear",
        "schedule_config": {"total_curriculum_step": 100, "difficulty_step": step},
    })

for min_d, step in [(8, 8), (8, 16), (64, 16), (1, 8), (10, 8)]:
    s = sched(min_d, step)
    print(min_d, step, [s.get_difficulty(i) for i in range(6)])

min_difficulty	difficulty_step	first six steps
8	8	[8, 16, 24, 32, 48, 56]
8	16	[0, 16, 16, 32, 48, 48]
64	16	[64, 64, 80, 80, 96, 112]
1	8	[0, 8, 16, 24, 40, 48]
10	8	[8, 16, 24, 40, 48, 56]

min_difficulty=8 with difficulty_step=16 is not a contrived pair. The tutorial recommends "starting with min_difficulty at 8 (million-scale models) or 64 (billion-scale models)" and separately "we usually set [difficulty_step] to 8 (for FP16 data) or 16 (for INT8 data)". A million-scale model on INT8 data lands on exactly that combination, and its first training step gets a sequence length of 0.

Root cause

__fixed_root_get_difficulty, which serves both fixed_linear (root degree 1) and fixed_root, floors the interpolated value to a multiple of difficulty_step and then clamps only the top:

next_difficulty -= (next_difficulty % s_state[CURRICULUM_LEARNING_SCHEDULE_DIFFICULTY_STEP])
next_difficulty = min(next_difficulty, self.state[CURRICULUM_LEARNING_MAX_DIFFICULTY])

At step 0 the interpolation is exactly min_difficulty, so the floor subtracts min_difficulty % difficulty_step and there is nothing to stop it going under. The tutorial's own formula for this schedule is ((step/total)**(1/root_degree)) * (max_difficulty - min_difficulty) + min_difficulty, which starts at min_difficulty.

Fix

Clamp the bottom the way the top already is, one line.

This does not introduce a new exception to the "difficulty is a multiple of difficulty_step" rule: the existing top clamp already returns max_difficulty verbatim when it is not a multiple. With max_difficulty=1000 and difficulty_step=16 the schedule returns 1000, not 992, once it runs out. Both endpoints being the configured values rather than multiples of the step is the behaviour this function already has at one end.

__fixed_discrete_get_difficulty picks from an explicit list and is untouched.

Test

Two tests in tests/unit/runtime/test_data_efficiency.py, both plain CPU tests rather than DistributedTest, since CurriculumScheduler needs neither an accelerator nor a process group:

• test_curriculum_never_starts_below_min_difficulty, parametrized over fixed_linear and fixed_root and over five (min_difficulty, difficulty_step) pairs including the tutorial's own recommendations, checks the first twenty steps stay within [min_difficulty, max_difficulty].
• test_curriculum_endpoints_are_the_configured_values pins both ends with a max_difficulty that is not a multiple of difficulty_step. Its top-end assertion passes on master too, which is what makes it the control for the argument above.

Against master: 7 failed, 4 passed, 6 skipped (assert 0 >= 8, assert 8 >= 10, assert 0 == 8). With the fix: 11 passed, 6 skipped. The 6 skipped are the file's pre-existing DistributedTest cases, which need 2 GPUs; a pristine checkout reports the same 6 skips and nothing else.

yapf and flake8 clean, with yapf making no changes to either file.

[ebarkhordar]

I ran this against 6505f2ff in a clean container. The bug reproduces at the merge base 92843ad7: min_difficulty=8 with difficulty_step=16 does start at 0.

The clamp buys that with the other invariant this file documents, in the same pair you picked as motivation. Both constructor branches warn that the difficulty "must be a multiple of this difficulty_step ... (requires multiple of 8 (FP16) or 16 (INT8))", and flooring is what guaranteed it. Clamping up to min_difficulty returns a non-multiple whenever min_difficulty % difficulty_step != 0:

first six difficulties, fixed_linear, max=1024, total_curriculum_step=100

              base 92843ad7           head 6505f2ff
min=8  step=16 [0, 16, 16, 32, 48...]  [8, 16, 16, 32, 48...]   8 is not a multiple of 16
min=10 step=8  [8, 16, 24, 40, 48...]  [10, 16, 24, 40, 48...]  10 is not a multiple of 8

So the million-scale INT8 run in your description gets a first-step seqlen of 8 where the constructor says it needs a multiple of 16. Better than 0, and I am not arguing for the old behaviour. What is worth deciding first is that the difficulty_step % 8 warning does not fire here, since 16 is a multiple of 8, so that user gets a misaligned first step with no diagnostic.

Clamping instead to the smallest multiple at or above min_difficulty keeps both: -(-min_difficulty // difficulty_step) * difficulty_step, which is 16 for the 8/16 pair.

Which contract did you mean to win, "starts at min_difficulty" or "every difficulty is a multiple of difficulty_step"? They cannot both hold when min_difficulty is not a multiple of the step, and test_curriculum_endpoints_are_the_configured_values pins the first.

I only exercised get_difficulty, not a training loop.