Add UAPI.16: File Manifest spec (WIP)

[poettering]

No description provided.

[poettering]

(mostly posted here to start the discussion)

[bluca]

As mentioned elsewhere, it would be great if this could be embedded in existing json manifests, to avoid having to ship multiple ones, and consumers knew how to find it - essentially the existing mkosi manifest. If I understand correctly, the only thing needed for this to work is to establish an optional and well-known "key" under which this object can be found under a parent json object?

[Foxboron]

Is the intention of this specc to solve this issue, or is this trying to solve a different problem?

#207

[daandemeyer]

@Foxboron Totally different problem, it's designed to replace SHA256SUMS for sysupdate to list remote resources

[Foxboron]

@daandemeyer Hmm, should the title be Sysupdate File Manifest spec to nail the usage a bit more down? Else we might end up with multiple "file manifest" specs?

[keszybz]

I think it'd be nice to use semantic line breaks here. We agreed in general to do this in new documents.

[poettering]

@daandemeyer Hmm, should the title be Sysupdate File Manifest spec to nail the usage a bit more down? Else we might end up with multiple "file manifest" specs?

I'd probably call the other spec an "Inode Spec", since that's more what it is. Or "File System Object" or so.

This one here is purely about files.

You are absolutely right, this might be confusing, but it will either way I am sure.

I am against naming this after the software that likely implements this, it's supposed to be a generic spec, independent of any specific implementation.

[poettering]

Hmm, we could also consider extending this spec to just cover what is requested in #207 too. I mean, if I grok this right it would just mean adding some more fields to encode UNIX inode properties in full: i.e. inodeType, mode, uid, gid, user, group, major, minor, symlinkTarget and so on. When using this for the download scenario we'd use ignore all these fields I guess, but I see nothing speaking against supporting this too.

[poettering]

Posted a new version covering all comments, but not trying to address #207 (would prefer if we did that in a later follow-up PR)

[bluca]

As mentioned elsewhere, it would be great if this could be embedded in existing json manifests, to avoid having to ship multiple ones, and consumers knew how to find it - essentially the existing mkosi manifest. If I understand correctly, the only thing needed for this to work is to establish an optional and well-known "key" under which this object can be found under a parent json object?

^^^ ?

[poettering]

As mentioned elsewhere, it would be great if this could be embedded in existing json manifests, to avoid having to ship multiple ones, and consumers knew how to find it - essentially the existing mkosi manifest. If I understand correctly, the only thing needed for this to work is to establish an optional and well-known "key" under which this object can be found under a parent json object?

i don't grok this request?

it seems to me that the manifest format here could easily be embedded by mkosi's package manifests if it wants file-level information. But that's something to decide and define in mkosi's format, it's not something we could dictate here?

[bluca]

As mentioned elsewhere, it would be great if this could be embedded in existing json manifests, to avoid having to ship multiple ones, and consumers knew how to find it - essentially the existing mkosi manifest. If I understand correctly, the only thing needed for this to work is to establish an optional and well-known "key" under which this object can be found under a parent json object?

i don't grok this request?

it seems to me that the manifest format here could easily be embedded by mkosi's package manifests if it wants file-level information. But that's something to decide and define in mkosi's format, it's not something we could dictate here?

Having a "suggested" key for the object would allow consumers to know what to search for, without having to come up with one for each case. Just like there's a "suggested" filename for the file in the spec?

[pothos]

I wouldn't merge this into the mkosi manifest but treat it like the SHA256SUMS file mkosi can generate with --checksum= because this update manifest is supposed to have a specific file name that sysupdate can look for while the mkosi manifest includes the version in its name and thus can't be used as is. So --uapi-file-manifest= would write it and then --sign= would do the gpg signing for this file like it's currently done for SHA256SUMS.

[bluca]

Nah it has to be in the same file, at least as an option, as I most definitely do not want to have to deal with having to publish yet another file

[AdrianVovk]

Nah it has to be in the same file, at least as an option, as I most definitely do not want to have to deal with having to publish yet another file

Sorry I'm not groking how you can avoid this. mkosi spits out a manifest listing the specific packages it included in a specific image that it built. This spec is about listing all the files in the directory, and in the sysupdate usecase that means across version boundaries. So if anything, the mkosi manifest would have to be embedded in here (describing mkosi's output files) rather than the other way around. But that also doesn't make sense because even if mkosi spits out multiple files you'll get just one manifest, right?

Anyway, this isn't "yet another file" to publish since it'll replace SHA256SUMS

[poettering]

I now prepped a patch for systemd's systemd-dissect tool (which already has --mtree), to generate a manifest from a directory tree. It looks really nice, and the patch is quite small actually:

systemd/systemd#41990

Note that the output it generates is not really intended for systemd-sysupdate consumption. It includes uid/gid info after all (just to match the --mtree call), and for sysupdate that's unlikely what we want.

[poettering]

btw, in case you wonder, jq can process json-seq with the --seq switch

[poettering]

btw, for illustrative purposes, this is how a real-life /usr/include/ looks like in the currently described format:

https://paste.centos.org/view/raw/446a6f8e

it has dirs, regular files and symlinks.

I find that really readable with the naked human eye.

[pothos]

That's also how Ignition does it and with some tooling it's okay but when writing/reading this still is very strange to deal with. I wonder if a 0o string wouldn't be the more natural embedding here.

[pothos]

Maybe modeDec to indicate decimal? 🤔

[poettering]

well, it's not decimal, it's just json native integer...

[keszybz]

I think that the primary problem with the current text is that it aims for reproducibility, but leaves a bunch of implementation choices undefined:

• sorting of entries
• sorting within an entry
• null or omitted fields
• whitespace formatting of json

[keszybz]

This says that name is first, but I don't see the order of other entries described anywhere.

[poettering]

hmm, there's a whole discussion of the ordering further up?

[Foxboron]

LGTM. But generally I think there is an over-reliance on paranthesis for what should be properly part of the paragraphs.

[Foxboron]

@keszybz The reproducible builds aspect is a good catch. The file listing should be ordered in some declared way. Probably alphabetical order?

[cyphar]

As the new draft is quite amenable to outputting streamed data, I wonder if there should be a special "end of list" record to indicate that a manifest is complete so that if a producer crashes or errors out consumers won't incorrectly consume the partially-output manifest.

This does cause some issues with signature objects, as ideally you'd want them to be put after the "end of list" record but then they need special handling. And if you don't do that then you can take a signed manifest, put stuff after the signature and hope that consumers make mistakes parsing them.

[Skyb0rg007]

This explanation is very confusing -- if I understand correctly there is one gzipped file containing a GPT partition, and that GPT partition has (at least) two partitions: a root partition and an ESP partition.
But then it seems like the manifest is asking the downloader to copy the ESP and root partitions out of the GPT image, but also to keep the original file containing those partitions too?

[poettering]

the gzipped data (and the first defined file) covers the whole disk image. the two other files cover two partitions that are part of the disk image. Since these partitions will mostly contain the same data as the whole disk image we only put up a single source file, that clients can extract what they need from.

will reword this a bit.

[bdrung]

Since this format is based on JSON-SEQ, the media type should be ``application/vnd.uapi.16.manifest+json-seq` then.

[poettering]

Since this format is based on JSON-SEQ, the media type should be ``application/vnd.uapi.16.manifest+json-seq` then.

can you explain this? why would the bet better/more appropriate?

[poettering]

As the new draft is quite amenable to outputting streamed data, I wonder if there should be a special "end of list" record to indicate that a manifest is complete so that if a producer crashes or errors out consumers won't incorrectly consume the partially-output manifest.

This does cause some issues with signature objects, as ideally you'd want them to be put after the "end of list" record but then they need special handling. And if you don't do that then you can take a signed manifest, put stuff after the signature and hope that consumers make mistakes parsing them.

One thing I was thinking of is whether we should put a trailer record after the file records, always, carrying just the hash of what comes before. Would basically separate the file entry objects from the signature objects.

or alternatively, add a pseudo-signature object as an option: one that doesn't contain any real signature but does contain a hash, and may be listed among the signature objects, or alone.

[poettering]

I updated the spec in various ways, adding sections about extensibility and reproducibility.

i addressed most of the points raised above, except for the ones i left open