Scratch: NVFP4 activation input_scale calibration study

[cjluo-nv]

What does this PR do?

Type of change: research artifact (in scratch/, not for merge)

Studies how the choice of input_scale affects NVFP4 activation
quantization MSE, on:

• synthetic distributions — including two pathological cases where
  more calibration hurts (a rare 1e6 spike in 1e-7 of samples; a
  log-normal σ=3 distribution whose amax never converges); and
• real Qwen3.5-9B MLP-input activations captured under the
  Nemotron-family calibration datasets used by modelopt's hf_ptq.py
  (chat-only, cnn_nemotron_v2_mix, nemotron-post-training-v3).

Three questions answered

1. Does more calibration always help?
   In principle, no — synthetic stress tests show 2.4 dB / 0.8 dB SNR
   losses when calibration captures rare outliers that inference
   doesn't see. In practice on Qwen3.5-9B + Nemotron-family data:
   calibration converges by ~500 sequences and bigger calibration is
   harmless (≤ 0.01 dB change up to N=2048).

2. Should we always use amax to derive input_scale?
   Yes — amax is within 0.01–0.06 dB of the MSE-optimal oracle on
   every layer × dataset combination tested. Percentile calibration
   (p99 / p99.9 / p99.99) under-shoots inference-time outliers and
   loses 1–25 dB SNR; only p99.999 matches amax.

3. Does calibration dataset choice matter?
   No — with a fixed held-out test tensor, cnn_nemotron_v2_mix and
   nemotron-post-training-v3 produce input_scale values within ~5%
   of each other, translating to ≤ 0.013 dB SNR spread on both
   measured layers.

Usage

# Synthetic distributions
python scratch/nvfp4_activation_calib_mse.py

# Real activations (modelopt PTQ default combo)
python scratch/capture_qwen35_mlp_activations.py \
    --n_seqs 2600 --max_tokens 512 --dataset cnn_nemotron_v2_mix
python scratch/nvfp4_real_activation_calib_mse.py

Testing

• All scripts run end-to-end on a single GPU (synthetic ~3 min,
  capture ~12 min, sweep ~1 min on RTX 6000 Ada).
• Pre-commit hooks (ruff, mypy, markdownlint, license headers) pass.
• No changes to library code — strictly additive under scratch/.

Notes

This is exploratory. The captured .pt activation tensors (~17 GB)
are excluded from the commit; everything in the report is
reproducible via the included scripts. Full discussion of methodology,
per-dataset sweep tables, fixed-test cross-combo comparison, and
limitations are in scratch/nvfp4_activation_calib_report.md.

Before your PR is "Ready for review"

• Is this change backward compatible?: ✅ (strictly additive under scratch/)
• If you copied code from any other sources or added a new PIP dependency: N/A
• Did you write any new necessary tests?: N/A (research artifact)
• Did you update Changelog?: N/A (scratch/ — not part of the library)
• Did you get Claude approval on this PR?: N/A (this is a research draft)

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8ced1814-b5c8-46ef-9789-4e0ec8c1ae51

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch chenjiel/nvfp4-activation-calib-study

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.49%. Comparing base (c9098b6) to head (c32f97f).
⚠️ Report is 11 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1545      +/-   ##
==========================================
- Coverage   76.75%   76.49%   -0.27%     
==========================================
  Files         476      477       +1     
  Lines       51811    52819    +1008     
==========================================
+ Hits        39767    40403     +636     
- Misses      12044    12416     +372     

Flag	Coverage Δ
unit	52.76% <ø> (+0.13%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.