Add configurable futures roll date

[scarab-systems]

Summary

• Adds DataMappingMode.TradingDaysBeforeExpiry for continuous futures that need to roll before the normal last-trading-day mapping date.
• Threads a tradeable-day roll offset and optional contract month cycle through continuous future subscriptions, universe settings, history requests, mapping events, and the Python wrapper.
• Reuses existing LastTradingDay map-file rows for the new mapping mode and applies the optional contract month cycle when walking continuous future contract depth.

Motivation

Fixes #9440.

The issue describes two related continuous futures needs:

• rolling a configurable number of trading days before the normal last-trading-day roll;
• restricting held futures contracts to selected contract months, for example skipping weak open-interest months.

Testing

• dotnet build Tests/QuantConnect.Tests.csproj --no-restore -clp:ErrorsOnly --verbosity quiet

Added focused tests for:

• TradingDaysBeforeExpiry reusing LastTradingDay map-file rows;
• contract-depth walking with a selected contract month cycle;
• skipping a current mapped contract when its month is outside the selected cycle;
• adding tradeable days across closed exchange dates.

I was able to build the test project locally. Direct local NUnit execution in my Linux container aborted during global Python/test-host initialization before the selected tests ran, so CI should be the authoritative full test run.

[brandon-68]

Dear All,

To provide some context, I actually reached out to Alex from QC team via QC ticket and gave him the draft specs for the proposed changes before he raised #9440, after which i provided a very comprehensive follow-up comment in #9440 detailing the proposed changes. So, I hope that #9538 would eventually land.

Coming back to #9538, the mapping-side design looks right to me: shifting the map-file search date forward by the tradeable-day offset, applied consistently in both ContinuousContractUniverse (drives the traded contract) and MappingEventProvider (drives data splicing), and the month-cycle walk in AdjustSymbolByOffset keeps SubscriptionDataConfig.MappedSymbol coherent since the getter returns the adjusted underlying's ID.

However, price normalization was not threaded through, and the failure is silent:

1. Common/Data/Auxiliary/MapFile.cs:108 aliases TradingDaysBeforeExpiry → LastTradingDay for mapping, but there is no equivalent for factors: Common/Data/Auxiliary/MappingContractFactorProvider.cs:66,75,84 match factor rows with row.DataMappingMode == dataMappingMode.
2. Factor files only contain rows for modes 0–3. This is verifiable in the repo's own shipped test data: every row in Data/future/cme/factor_files/es.csv carries "DataMappingMode":0/1/2/3. With mode 4, no row ever matches and GetPriceFactor falls through to the default — 1 for BackwardsRatio, 0 for the Panama modes.
3. Both scaling paths pass the config's mapping mode: Engine/DataFeeds/SubscriptionUtils.cs:132 and Engine/DataFeeds/Enumerators/PriceScaleFactorEnumerator.cs:112.
4. BackwardsRatio is the default normalization for futures (Common/Extensions.cs:3973), so any user adding a continuous future with the new mode gets an unadjusted stitched series with roll gaps — no error, no warning. Every indicator and backtest statistic computed on it would be contaminated.

Note that simply aliasing to LastTradingDay factor rows would not be correct either: with rolls shifted N tradeable days earlier, the splice factor must be computed at the shifted roll date, and since N is a free user parameter, factors cannot be precomputed per (mode, N). Correct support seems to require either on-the-fly factor computation at mapping events from the two contracts' prices, or changes to the data pipeline — I'd welcome the QC team's view on which is preferred.

Suggestions to make this mergeable:

• Short term / fail fast: in MappingContractFactorProvider.GetPriceFactor, throw or at minimum log a warning when the mode is TradingDaysBeforeExpiry and an adjusted normalization mode is requested, rather than silently returning the default factor.
• Regression algorithms: the repo convention for continuous-futures behavior changes (cf. ContinuousFutureRegressionAlgorithm.cs) is C#/Python regression algorithms with pinned statistics. I'm attaching a skeleton that asserts (a) rolls occur exactly dataMappingModeDaysOffset tradeable days early, (b) the newly mapped contract has more than the offset remaining, and (c) the BackwardsRatio series is genuinely price-adjusted across rolls — assertion (c) will currently fail and is the detector for the gap above.
• Secondary: MappingEventProvider.Initialize now calls MarketHoursDatabase.FromDataFolder().GetEntry(...) unconditionally for every mapped subscription, equities included; consider resolving lazily (only when the mode is TradingDaysBeforeExpiry) to avoid a new hard MHDB dependency on previously working subscriptions.

Happy to help test. This PR implements #9440, currently among the top-voted items on the QC roadmap.

ContinuousFutureTradingDaysBeforeExpiryRegressionAlgorithm.cs

[brandon-68]

Local test results confirming the review above (commit 44003ff)

Environment: Windows, .NET SDK v 10.0.301, LEAN v2.5.0.0, fresh clone with pull/9538/head checked out, repo-shipped sample data only. Unit tests were not run locally (Tests/AssemblyInitialize.cs initializes an embedded Python runtime for the whole assembly regardless of test filter, requiring the foundation-container Python 3.11 environment — possibly the same wall the PR description mentions hitting); deferring unit test validation to CI.

Method: matched A/B backtest using the regression algorithm below (ES continuous contract, 2013-07-01 → 2014-01-01, BackwardsRatio, monthly trading). The two runs are identical except for the mapping mode; the algorithm asserts (1) rolls fire dataMappingModeDaysOffset tradeable days early, (2) the newly mapped contract has more than the offset remaining, (3) the streamed continuous series is genuinely back-adjusted vs. the raw mapped contract price.

### Run A — TradingDaysBeforeExpiry, offset 2 (full log attached)
runA_TradingDaysBeforeExpiry_clean.txt

Roll timing works. Mapping events fired on 2013-09-18 and 2013-12-18 — exactly 2 tradeable days earlier than the control's 09-20 / 12-20. Assertions (1) and (2) passed. The PR's core mapping mechanism is validated.

The series is not price-adjusted. Across 26,670 compared bars, max |adjusted/raw − 1| = 0.000624 — the continuous price equals the raw contract price on every bar:

2013-10-07 09:31:00 - adjusted continuous close: 1669.5, raw ES VMKLFZIH2MTD: 1669.375, ratio: 1.000075
2013-11-01 09:31:00 - adjusted continuous close: 1700, raw ES VMKLFZIH2MTD: 1699.625, ratio: 1.000221
2013-12-02 09:31:00 - adjusted continuous close: 1806.25, raw ES VMKLFZIH2MTD: 1806.375, ratio: 0.999931
Runtime Error: BackwardsRatio continuous series is NOT price adjusted: streamed continuous
price equals the raw mapped contract price (max deviation 0.000624). No factor rows exist
for DataMappingMode.TradingDaysBeforeExpiry, so MappingContractFactorProvider.GetPriceFactor
returned the default factor (1.0).

A second run reproduced the failure identically (same max deviation to six decimal places).

### Run B — control, LastTradingDay, same algorithm/data (full log attached)
runB_LastTradingDay_control_clean.txt

Completes cleanly. Ratios ~0.9346 on every sampled bar (max deviation 0.065880), matching the shipped Data/future/cme/factor_files/es.csv factors:

2013-10-07 09:31:00 - adjusted continuous close: 1560.27..., raw ES VMKLFZIH2MTD: 1669.375, ratio: 0.934647
2013-11-01 09:31:00 - adjusted continuous close: 1588.78..., raw ES VMKLFZIH2MTD: 1699.625, ratio: 0.934783
2013-12-02 09:31:00 - adjusted continuous close: 1688.07..., raw ES VMKLFZIH2MTD: 1806.375, ratio: 0.934512

### Scope of the bug

Notably, OrderListHash is identical between the two runs (1973b0beb9bc5e618e0387d960553d7a) — orders fill on raw contract prices, so trading mechanics are unaffected. The corruption is confined to the continuous data series: every indicator, signal, and statistic computed on it under the new mode would silently use unadjusted prices with roll gaps.

(Note: the 95% failed data requests in the DataMonitor section is expected with the sparse repo sample data and unrelated to the finding.)

Regression algorithm used

ContinuousFutureTradingDaysBeforeExpiryRegressionAlgorithm.cs

Once the normalization gap is fixed, this algorithm should pass under both modes; the
ExpectedStatistics/DataPoints placeholders can then be pinned and a Python twin added
to make it a standard regression test.

[scarab-systems]

Thanks for the detailed review and local repro. I agree that the adjusted-normalization path should not silently fall through to default factors for TradingDaysBeforeExpiry.

I pushed cb09f22c8 to make that failure explicit for now:

• MappingContractFactorProvider.GetPriceFactor now rejects TradingDaysBeforeExpiry with adjusted continuous-future normalization, because the existing factor files do not contain shifted-roll-date factors for that mode.
• Raw normalization remains unaffected.
• Added factor-provider regression coverage for BackwardsRatio, BackwardsPanamaCanal, and ForwardPanamaCanal so this cannot silently regress back to default 1/0 factors.

Verified in the Lean foundation container:

dotnet test Tests/QuantConnect.Tests.csproj --filter "FullyQualifiedName~RejectsTradingDaysBeforeExpiryForAdjustedFutureFactorFiles" -clp:ErrorsOnly --verbosity quiet
dotnet test Tests/QuantConnect.Tests.csproj --filter "FullyQualifiedName~FactorFileTests" -clp:ErrorsOnly --verbosity quiet

Both passed. This keeps the shifted-roll mapping path honest while leaving the larger question of dynamic shifted-roll factor computation/data-pipeline support for QC guidance.

[brandon-68]

@scarab-systems : Thanks for the swift response! Much appreciated! Here are further testings and findings from me. Hopefully they can be of help.

Follow-up on cb09f22: guard verified (with one placement finding) + contractMonthCycle empirically validated

Guard check. Re-ran the earlier A/B harness (mode 4 + BackwardsRatio) against cb09f22.
The guard fires correctly — all three of the continuous future's subscriptions (trade, quote,
open interest) throw the intended exception at their first price bar:

guard_run_cb09f22_clean.txt

ERROR:: Subscription worker task exception ES20Z13,#0,ES VMKLFZIH2MTD,Minute,TradeBar,Trade,BackwardsRatio,TradingDaysBeforeExpiry. System.NotSupportedException: TradingDaysBeforeExpiry does not support BackwardsRatio normalization because continuous future factor files do not contain factors for shifted roll dates. Use Raw normalization or choose a factor-file-backed mapping mode. at QuantConnect.Data.Auxiliary.MappingContractFactorProvider.GetPriceFactor(...) in Common/Data/Auxiliary/MappingContractFactorProvider.cs:line 48 at QuantConnect.Data.Auxiliary.PriceScalingExtensions.GetPriceScale(...) in Common/Data/Auxiliary/PriceScalingExtensions.cs:line 76 at ...SubscriptionUtils.<>c__DisplayClass1_0.<CreateAndScheduleWorker>b__0(...) in Engine/DataFeeds/SubscriptionUtils.cs:line 132
One placement finding: the exception is swallowed by the subscription worker's catch-all
(SubscriptionUtils.cs:154, catch (Exception) { Log.Error(...) }), which stops the affected
subscriptions but lets the algorithm run to completion. A user hitting this config therefore gets
a flat, zero-order backtest with only log-level ERRORs — no runtime error, and downstream mapping events stop flowing once the workers die. Suggest raising the check at subscription-creation time instead, mirroring the existing precedent for unsupported normalization combinations: the ScaledRaw guard in Engine/DataFeeds/DataManager.cs:305 (a file this PR already modifies) and the options Raw-only check in Common/Securities/Option/Option.cs:629. That would fail fast in Initialize with a clear message.

contractMonthCycle verification. Extended the observation period to Jul 2013 – Feb 2016
(ES, Raw normalization, repo-shipped data; logs attached) across four configurations:

#	Configuration	Result
1	LastTradingDay, depth 0 (control)	Quarterly front-month rolling, 10 rolls at each contract boundary — baseline behavior intact
2	+ contractMonthCycle: {6,12}	Holds only Jun/Dec contracts from initialization (Sep'13 skipped); 5 semi-annual rolls, each at the held contract's own boundary; out-of-cycle boundaries produce no spurious events
3	+ contractDepthOffset: 1	Tracks the second-nearest Jun/Dec contract; rolls on identical dates to #2, always one cycle-contract apart — #2 and #3 form a synchronized calendar pair, the property a spread/carry position needs
4	#3 + TradingDaysBeforeExpiry, offset 2, Raw	Identical contract sequence to #3, every roll exactly 2 tradeable days earlier (12-18 vs 12-20, 06-18 vs 06-20, 12-17 vs 12-19, 06-17 vs 06-19, 12-16 vs 12-18)

variation1_clean.txt
variation2_clean.txt
variation3_clean.txt
variation4_clean.txt

With this, the full mapping-side feature set of #9440 (month cycle, depth within cycle, shifted
roll) is verified over 2.5 years, individually and combined. Remaining scope unchanged: factor
generation for shifted rolls (with QC), the guard placement above, then regression algorithms
with pinned statistics. QC's guidance on the factor approach has been requested through a support
ticket in parallel.

[scarab-systems]

Thanks again for the detailed follow-up, and apologies for the delay in responding. We wanted to take the time to check the failure boundary carefully rather than just leave the guard where it was.

We pushed eea3e0fb8 to move the unsupported adjusted-normalization check into DataManager.AddSubscription, next to the existing ScaledRaw guard. So TradingDaysBeforeExpiry with adjusted normalization now fails during subscription setup before the subscription worker is scheduled, while the provider-level guard remains as a backstop.

We left the mapping-side behavior unchanged given your verification of month cycle, depth, and shifted Raw rolls.

We also looked into the remaining shifted-roll factor-generation side. We did not include that in this update because it seems like the right implementation should follow QC's preferred factor approach, but if QC would like us to take that on, we're happy to provide the follow-up implementation.

Validation in quantconnect/lean:foundation:

• TradingDaysBeforeExpiryAdjustedNormalizationFailsBeforeCreatingSubscription: passed, 3 cases
• RejectsTradingDaysBeforeExpiryForAdjustedFutureFactorFiles: passed, 3 cases
• dotnet build Tests/QuantConnect.Tests.csproj: succeeded