BUG: Size shared memory by materialized array, not lazy Index.nbytes - #1385
Merged
Conversation
RangeIndex is lazy: its .nbytes reports the size of the range object itself (a constant 132 bytes), not the int64 array it materializes into. SharedMemoryManager.arr2shm sized the block with vals.nbytes and then built an ndarray of vals.shape/vals.dtype.base over the buffer, so Backtest.optimize() on range-indexed data raised 'TypeError: buffer is too small for requested array' while bt.run() on the same data worked. Size the block from the same shape/dtype pair the buffer view is built with. For eager indexes (Index[int64], DatetimeIndex, tz-aware) the expression equals .nbytes, so only the broken RangeIndex path changes. Fixes kernc#1237
Owner
|
Excellent work! Many thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1237.
The bug
Backtest.optimize(method='grid')fails withTypeError: buffer is too small for requested arraywhenever the data has a RangeIndex, whilebt.run()on the same data works fine.Root cause in
SharedMemoryManager.arr2shm(backtesting/_util.py): the shared-memory block is sized withvals.nbytes, but RangeIndex is lazy — its.nbytesreports the size of the underlyingrangeobject itself (a constant 132 bytes), not the int64 array it materializes into. The very next line buildsnp.ndarray(vals.shape, dtype=vals.dtype.base, buffer=shm.buf), which needs8 × len(index)bytes. Measured:pd.RangeIndex(44640).nbytes == 132vs 357 120 bytes required — so any range-indexed frame longer than 16 rows overflows.Contrary to the issue title, size is not the real variable — index type is. The "under 10 000 rows works" observations in the thread were almost certainly datetime-indexed runs, and range-indexed data is an explicitly documented input (
Backtest.__init__: "a monotonic range index (i.e. a sequence of periods)").The fix
Size the block from the same
shape/dtype.basepair the buffer view is built with:vals.size * vals.dtype.base.itemsize. For every eager index in play (Index[int64],DatetimeIndex, tz-awareDatetimeIndex, float Series) this equals.nbytes, so only the broken RangeIndex path changes behavior.Validation
test_optimize_range_index(placed next to the closely-relatedtest_optimize_datetime_index_with_timezone): red on master with the exactTypeErrorfrom the issue, green with the fix.multiprocessing.Pool(not the Windows thread fallback), a 4-point grid on range-indexed GOOG returns the same best equity (81 812.37) as four independently computed serialbt.run()calls.python -m backtesting.test: 81 tests, OK (1 skipped).flake8: clean.mypy: error list byte-identical to master (all pre-existing, none introduced).