tiles: add Frontier to read/write tiles - #8896
Conversation
Frontier is loaded at MTCA startup, tracks appended entries and hash tiles, and writes them to storage. Implementation is written by me. The unittest in tiles_test.go was written by Claude and reviewed and lightly edited by me (including rewriting the comments in my own words).
|
FYI to reviewers: I was going to implement RootHash in another branch, but it wound up being quick enough I decided to include it in this one. Also, in implementing RootHash it became clear I should switch up the design to allow empty tiles on the frontier and forbid full ones (instead of the other way around). So I did that refactoring and everything got a little bit nicer. |
ezekiel
left a comment
There was a problem hiding this comment.
It is cool to see this pull together the entry Bundles and factored-out S3 code to get to the point of a really well-structured and readable tiles package.
Just two tiny comments to consider from me.
|
Pushed responses to most; will think about where to put |
|
Note: this PR doesn't yet implement recovery from partial flushes. The general sketch of of that will work is to imitate FiloSottile/sunlight@a481639: write once with a prefix pointing to a staging area. Then sign and store a checkpoint. Then copy from the staging area to live storage. Then publish the checkpoint. See #8902. |
beautifulentropy
left a comment
There was a problem hiding this comment.
This change looks great, I have just one additional test I'd like but it also requires a small fix to the S3 mock.
Frontier is loaded at MTCA startup, tracks appended entries and hash tiles, and writes them to storage.
Implementation is written by me. The unittest in tiles_test.go was written by Claude and reviewed and lightly edited by me (including rewriting the comments in my own words).
Move subtree's
mth()function out from the unittest and export it, since it's needed to calculate a hash to append to a parent tile once its child tile becomes full.Fixes #8894