Skip to content

Support Restatements on SCD Type 2 and Use Them When Creating Preview Clone #5901

Description

@StuffbyYuki

Currently SCD Type 2 supports only restating the model from the beginning of time. One could also restate a model at any point in history too but also end with the latest date. This how how it would work:

Lets say I am restating Jan 10th

Delete from the table where all valid_from >= Jan 10th
Update the table where all valid_to >= Jan 10th
This would reset the table to look like it was last run before the given start date.

Ideally, once this is implemented, when we create a preview clone of a model we would then configure the clone to restate this latest model with the new logic.

Suggestion for resolving that is to update the SCD Type 2 logic to always go through this delete/update logic on each insert. That way in most cases where we are just processing the newest interval it would be no-op (does require a scan though) and when we have to restate it would delete/update as expected.

Inherited from #3642

Metadata

Metadata

Assignees

No one assigned

    Labels

    FeatureAdds new functionalityImprovementImproves existing functionality

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions