Currently, migrateDataDump/migrateDataDump' works by first validating the changes, which computes a list of all the APITableChanges, then by applying the changes to the data. That means it retains all the intermediate NormAPIs, which may be large (albeit probably with quite a lot of sharing).
In principle, migration could work in a single pass over the changelog, computing the new API and data at each step. Of course, that might do unnecessary work if the changelog turns out to be invalid, but in the case of a valid changelog with many entries, it would be better to use a single-pass approach as then the intermediate APIs are not retained longer than necessary.
It isn't completely obvious how much benefit this would give, as sharing may mean the marginal cost of storing the intermediate APIs is low, but I thought it was worth writing down in case we want to look at migration performance in the future. Certainly the cost of serialising the intermediate APIs can be very high (see #85), but that shouldn't normally happen anyway.
Currently,
migrateDataDump/migrateDataDump'works by first validating the changes, which computes a list of all theAPITableChanges, then by applying the changes to the data. That means it retains all the intermediateNormAPIs, which may be large (albeit probably with quite a lot of sharing).In principle, migration could work in a single pass over the changelog, computing the new API and data at each step. Of course, that might do unnecessary work if the changelog turns out to be invalid, but in the case of a valid changelog with many entries, it would be better to use a single-pass approach as then the intermediate APIs are not retained longer than necessary.
It isn't completely obvious how much benefit this would give, as sharing may mean the marginal cost of storing the intermediate APIs is low, but I thought it was worth writing down in case we want to look at migration performance in the future. Certainly the cost of serialising the intermediate APIs can be very high (see #85), but that shouldn't normally happen anyway.