Core architecture + DB persistent storage; PFS for one-to-one messages; refactoring and internals redesigns#90
Open
jagerman wants to merge 81 commits intosession-foundation:devfrom
Open
Conversation
session-deps is a new repo that consolidates how we handle loading and doing static bundle builds of various common external dependencies across Session projects.
Carrying a libsodium fork is too much of a nuissance as updating it to the latest version is non-trivial. This drops the libsodium-internal fork in favour of using tweenacl's implementation *just* for the X25519 -> Ed25519 pubkey conversion, and using a stock libsodium for everything else. This also bumps the libsodium requirement up to 1.0.21: that version will be required for SHAKE support in future commits on this branch.
- Blake2b hashing:
- Add nicer hash::blake2b functions for simpler has computations where
you can just pass a bunch of spannables and get the hash over them,
rather than needing to do a bunch of manual C API update calls.
- Add a ""_b2b_pers for a compile-time validated blake2b
personalisation string
- Drop make_blake2b32_hasher().
- TODO: convert these to make full use of hash::blake2b(...), as
above.
- Change cleared_array to take a Char type instead of forcing unsigned
char.
- Add cleared_uchars (and cleared_bytes) for the old force-unsigned char
typedef.
- Make `to_span` work for any input convertible to a span
- Tighten up various functions taking fixed length values (like session
ids, pubkeys) to take compile-time-fixed spans instead of dynamic
extent spans. This allows these generic functions to not have to
worry about length checking.
- Add a generic random::fill(s) function that fills some spannable type
s with random bytes.
This switches to the PR branches for session-router and libquic to use session-deps. (This was required under ninja builds, in particular, to get the deduplication handling for gmp and nettle via new session-deps code to deal with that).
- Refactor manual libsodium blake2b hash calls to use simpler hash::blake2b functions instead. - Unify usage of array_uc32/33/64 and uc32/33/64: now we have just uc32/33/64 and cleared_uc32/33/64 (i.e. the "array_" prefix is gone). - Add a unsigned char array literal, which is quite useful for static hash keys.
- Including raw integer bytes would break the hashes on a non-little endian arches; this adds a helper than ensures we are always hashing the little-endian value. - Make the remaining manual hash use blake2b_pers. This didn't get autoconverted before because the `if` around part of the hash, but that if is actually completely unnecessary: if the value is empty, a blake2b hash update does nothing, and so it can just be always included (and when empty, it is still the right thing).
oxenc has buggy "constexpr" overloads that just break if invoked, and they get invoked here with a uint8_t value + unsigned char array. The issue is fixed in oxenc dev, but switching to a byte here works around it for current and older oxenc versions.
The recent commit to unify the types wasn't applied to the test suite.
hash::blake2b (and related) now take integers directly, writing the integer bytes (with byte swapping applied, if necessary), which further simplies the hashing API.
Make way for account keys to be in here as well.
If two devices rotate account keys at approximately the same time, both might end up with inconsistent "active" keys. This commit adds deterministic tie breaking (always prefering the later, with fallback to seed ordering).
Adds a "needs push" concept, along with more tracking fields to let us distinguish between various possible states.
`session::AdjustedClock` now carries an adjustable static offset and when `now()` is called it returns standard system clock timepoints with the adjustment applied. The networking code had a similar adjustment already, although it was per-network-object: this change replaces that, and makes it now global across all uses of the clock anywhere, instead of per-Network instance.
This refactors the code taking dynamic extent spans to enforce that only compile-time-known 32 or 64-byte values can be passed. It also adds an intermediate class to hold the span on its way as an argument, as well an auto-expander in that intermediate argument class.
Adds a new `tests/testLive` binary that carries its own test suite that runs tests on testnet via the session networking code, and is capable of using direct, onion request, or session-router requests (the latter currently doesn't work and needs further investigation, but is out of scope of this PR).
We write the test databases into /tmp, but that means running two test suites at once can clash and corrupt each other. This fixes it by using random::unique_id for the db filenames (which has a unique counter *and* a random value). Additionally the poll test was using a fixed filename, making it even more likely to break. This converts poll to use the same TempCore as the other various tests.
We were only keeping the Ed25519 secret key in Core::Globals's secure root key buffer, but we actually frequently need the X25519 private key as well to perform various message decryptions. This expands that buffer by 32 bytes (to 96) to pack the precomputed x25519 key in there as well -- previously we had to do an unlock + convert, but with this change we just unlock and have the converted value cached.
uint8_t is a typedef to unsigned char on all modern systems, but the *intent* is different: uint8_t says "I want an unsigned integer value", while "unsigned char" says "I want a consistent-signed byte value". This makes everything use the latter *except* for things that are specifically numeric values.
…ode functions
The encoding side of session_protocol previously used a runtime-dispatch
pattern: callers populated a Destination struct (C++) or
session_protocol_destination struct (C API) with a type tag
(DestinationType / SESSION_PROTOCOL_DESTINATION_TYPE_*) and a union-like
collection of fields, then passed the whole thing to a single
encode_for_destination / session_protocol_encode_for_destination
dispatcher that switched on the type tag to call the appropriate
internal logic.
This pattern had several problems:
- The struct carried fields for all destination types simultaneously,
making it unclear at the call site which fields were relevant for a
given type. Callers had to know implicitly that e.g. community_pubkey
is meaningless for a 1o1 message.
- The type-tag switch was a source of silent bugs: adding a new
destination type required updating the switch in multiple places, and
an unhandled case would silently fall through.
- The structs and enums leaked into the public API despite carrying no
semantic value beyond routing — they existed purely as a workaround
for having a single entry point.
- On the C++ side there was also a redundant _impl layer: four static
*_impl functions held the real logic, thin public wrappers forwarded
to them, and C API wrappers also forwarded to them — three levels of
indirection for no benefit.
Replacement: Four typed, direct encode functions, one per message kind:
- encode_dm_v1 / session_protocol_encode_dm_v1 — one-on-one and sync
messages (renamed from encode_for_1o1 to better reflect the protocol
layer)
- encode_for_community_inbox /
session_protocol_encode_for_community_inbox — blinded community DMs
- encode_for_community / session_protocol_encode_for_community —
community (open group) messages
- encode_for_group / session_protocol_encode_for_group — closed group v2
messages
Each function takes exactly the parameters it needs and nothing else.
The type system enforces correctness at the call site: there is no way
to accidentally omit a required field or supply an irrelevant one. The
_impl layer is gone; each public function directly contains its
implementation body, with encode_for_community_inbox delegating to
encode_for_community for the shared pad-and-optionally-sign logic.
The Destination, DestinationType, session_protocol_destination, and
SESSION_PROTOCOL_DESTINATION_TYPE_* types are fully removed from both
the C++ and C APIs. The single-dispatch encode_for_destination /
session_protocol_encode_for_destination entry points are removed along
with them.
Related: Ed25519PrivKeySpan / OptionalEd25519PrivKeySpan improvements
All four encode functions take the pro rotating key as const
OptionalEd25519PrivKeySpan&. Two constructor improvements were made to
these types to make call sites cleaner:
- Ed25519PrivKeySpan(const unsigned char*, size_t) is now non-explicit,
allowing {ptr, size} brace-init at call sites (safe with two
parameters — no silent single-argument conversion risk). The redundant
from(ptr, size) factory was removed.
- OptionalEd25519PrivKeySpan(const unsigned char*, size_t) was added as
a non-explicit constructor: size == 0 produces the null (no-pro)
state; size == 32 or size == 64 constructs the key. This allows C API
boundaries to pass {ptr, len} naturally without an explicit
OptionalEd25519PrivKeySpan{...} wrapper.
- Deprecate hash::hash -- we have several hashes in use and the more specific one makes more sense. - Convert various remaining places still using raw C API calls. - Add hash::blake2b (and so on) overloads that take a templated hash size, e.g. blake2b<64>(...) and return a hash into an std::array of that size rather than needing the caller to preconstruct such an array to pass it in.
This adds wrappers around the C functions we use from sodium & mlkem_native so that we can always go through the wrappers taking std::byte spans, giving us compile-time length requirements (where appropriate), collapsing `data, size` argument pairs into single arguments, and just making it much less painful to use std::byte everywhere because with these changes we won't have to use endless reinterpret_casts. This is part 1: it adds the wrappers needed to make it work.
This adds a encryption implementation *without* PFS that can be used if clients cannot find the recipient's PFS+PQ keys for some reason. Such decryption gets flagged as non-PFS (so it can be visually indicated, e.g. maybe with a yellow padlock or some such warning icon).
snprintf_clamped was a C-style variadic wrapper around vsnprintf that existed to work around snprintf's confusing return value semantics. Meanwhile, fmt::format_to_n does *exactly* what is needed, and fmt is available everywhere in our codebase. Plus you get type safety, compile-time format checking, modern formatting, and no varargs. snprintf_clamped should never have been here. fmt is faster, too. Aside from being worse in every way, the snprintf_clamped usage was buggy: all three call sites in pro_backend.cpp passed sizeof(result.error_count) for the clamping size, i.e. sizeof(size_t), i.e. 8 or sometimes 4), instead of sizeof(result.error) (i.e. 256), silently truncating every error message to 7 characters. The call sites also required C-isms like `"%.*s", static_cast<int>(s.size()), s.data()` just to copy a std::string into a buffer — a pattern that was copy-pasted across 13+ call sites. This replaces all the snprintf_clamped usage with two small helpers in internal-util.hpp: - copy_c_str: copies a string_view into a fixed char buffer, truncating and null-terminating (with a char[N] overload that deduces the size to simplify calling verbosity). - format_c_str: fmt::format_to_n into a fixed buffer with proper compile-time fmt format checking This also removes the bool-returning set_error(char*, exception&) overload (and its duplicate in session_network.cpp), replacing the callers with explicit copy_c_str + return statements. The bool return was invariant and was being used to put the actual function return value in some unrelated function.
This adds methods and callbacks to the Core object to support sending DMs through it, with "v2" direct messages used if the recipient has published PFS+PQ support (and thus v2 support). This supports both using a built-in Network object, and supplying an external callback to send the encrypted DM (which would be either v2 PFS, v2 no-pfs, or v1, depending on flags and whether PFS+PQ support was found). For both, the input is a serialized protobuf "Content", and the message that gets sent is padded, encrypted, and encoded as required (as v2 or v1), including (for v1) the extra 17 layers (approximately) of protobuf. This also fixes some bugs in the existing v2 helper methods that were taking a pre-existing Pro signature (which is impossible, because the caller doesn't know what to sign) and should instead take the key to sign with. Additionally fixed the new and existing API to take a proper sys_ms timepoint rather than abusing std::chrono::milliseconds as a timestamp. Adds unit tests for DM sending.
This adds support for the (still experimental) "quic-files" protocol for streaming uploads and downloads (see session-file-server PR session-foundation#7). This has several components: Session Network w/ Session Router upload/download ================================================= session::network when in session-router mode now has a quic file client backend that looks for known pubkey URLs and, if found, makes the upload or download request over session-router to a configured mainnet or testnet session-router address with a running quic-files server. Download URLs with Session Router addresses =========================================== Download URLs now support an optional sr= fragment for specifying a session-router address, such as: http://host/file/ID#sr=address.sesh:11235 (port is optional, 11235 is the default). This allows upgraded clients to include session-router addresses for custom file servers, while remaining backwards-compatible with older clients that ignore unknown fragments. Upload/download test script =========================== A new tests/quic-files target is built that allows command-line upload/download testing, using mainnet or testnet; direct connections or session-router. Live tests now using session-router =================================== The "liveTest" script now supports upload and download tests (on testnet), using direct, onion requests, and session-router. A new CI job is added that runs all three.
This adds a streaming encryptor: it still has to read the file twice to
be used (once to find the key, then once while encrypting), but it never
has to hold the entire file in memory at once.
Also while in there I found some awkwardness in the encryption keygen,
where we were using:
[key, nonce] = blake2(size=56, domain=[0|1], seed || content)
which is better constructed as:
[key, nonce] = blake2b(size=56, domain=seed, pers="...", content)
where pers is either SessionAttachmnt or Session_Prof_Pic, depending on
the attachment type.
This change *does* change all the encryption keys (and nonce) from what
they would be before this change, but doesn't affect decryption, and is
conceptually cleaner, so seems worthwhile.
This allows on-the-fly encryption and transfer to the big file server without ever needing to store the entire file in memory.
This allows catching C++ free functions that don't have declarations earlier, such as frequently happens during a refactor that changes arguments (and can leave the old function behind in the .cpp, unnoticed until linking fails to find the one declared in the header). Fixes caught by the new warning: - Add missing `static` qualifiers to internal-only functions across multiple source files - Add missing `#include <string_view>` in session_protocol headers - Add schema_migrations.hpp.in template for generated migration headers Other changes bundled in this commit: - Refactor pro_features_for_utf8/utf16: replace raw char*/size parameters with std::string_view and std::span<const std::byte> overloads; split the combined utf8-or-16 helper into separate validate-then-check functions - Add file_server::extend_ttl() request builder for extending file TTLs - Move compress_message declaration into base.hpp
Upload/download improvements: - Add on_progress callback to FileTransferRequest (upload and download), fired at most once per progress_interval (default 1s) and only when progress has been made; uploads report acked bytes, downloads report received bytes - Add stall_timeout (default 25s) to FileTransferRequest: kills the upload if no ack progress is seen for that duration - Disable the idle timer during streaming uploads; restart it afterward for connection reuse - Fix streaming upload to not send the preamble twice (was causing 453 "too_much_data" from the server) - Call on_progress with 100% on successful upload completion if the final acked-bytes tick hadn't reached total yet Session-router ready-state gating: - Add _pending_operations queue to SessionRouter (parallel to _pending_requests) for non-HTTP operations - Gate _upload_internal, _download_internal, and upload_file on _ready, queuing them for replay in _finish_setup if the router is not yet ready - Split upload_file into dispatch-to-loop + _start_file_upload() to enable safe queueing MTU cap (opt::quic_max_udp_payload): - Replace opt::quic_disable_mtu_discovery with opt::quic_max_udp_payload (keeps PMTUD active but caps it); deprecate old option as a subclass - Thread max_udp_payload through NetworkConfig, C API (session_network_config), QuicFileClient, and _get_file_client - Wire tunnel_info::suggested_mtu (from session-router) into all _get_file_client calls so the per-tunnel MTU cap is applied automatically quic-files CLI: - Default to testnet; add --mainnet flag - Default server pubkey/address from QUIC_FS_ED_PUBKEY_TESTNET / QUIC_FS_SESH_ADDRESS_TESTNET constants (no longer required for --direct) - Add --max-udp-payload flag - Route all user-facing output through fmt::print (stderr for status/progress, stdout for results); remove logcat - Use separate nodedb cache dirs for mainnet vs testnet - Add upload progress callback at 250ms interval showing speed/percentage - Add app.fallthrough() so unknown args fall through to subcommands
This adds a b32 typedef for an std::array<std::byte, 32>, intended to replace uc32. It also renames bytes32 to cbytes32 to avoid cognitive confusion as to what the two types are doing.
Reorganize cryptographic APIs into algorithm-specific namespaces under
session:: (ed25519, x25519, mlkem768) with headers grouped under
include/session/crypto/ and implementations in src/crypto/. Libsodium
and mlkem_native headers are now confined to .cpp wrapper files, keeping
public headers dependency-free.
Key changes:
- New include/session/crypto/{ed25519,x25519,mlkem768}.hpp with clean
std::byte APIs; implementations in src/crypto/*.cpp isolate all
libsodium/mlkem includes.
- ed25519::PrivKeySpan: non-copyable span-like type that accepts 32-byte
seeds or 64-byte keys, with automatic seed expansion. Uses
Ed25519KeySpannable concept (std::convertible_to, not
constructible_from) to prevent dynamic-extent span mismatches.
- Deleted include/session/crypto.hpp (consolidated into algorithm
namespaces) and include/session/curve25519.hpp (replaced by x25519.hpp
+ ed25519.hpp conversion functions).
- Split decode_envelope into decode_dm_envelope (Ed25519 DH for 1-on-1)
and decode_group_envelope (symmetric key for groups), eliminating type
confusion in the shared DecodeEnvelopeKey struct.
- Added cleared_vector<T> (vector with clearing_allocator that zeros on
dealloc) and implicit span conversions on sodium_array.
- Added named constants for xchacha20 and mlkem768 parameters.
- Extended hash::blake2b HashInput concept to accept raw std::byte.
Update merged branch code with std::byte refactor modifications.
Add include/session/formattable.hpp with a concept-constrained
fmt::formatter that works for any contiguous range of std::byte
(std::span, std::array, std::vector, etc.). This enables lazy formatting
of binary data in log calls — the encoding only happens if the log
message is actually emitted, unlike the previous pattern of eagerly
calling oxenc::to_hex() before the log call.
The byte_spannable concept is restricted to std::byte element types only
(not unsigned char) to avoid conflicts with built-in fmt formatters and
to align with the ongoing unsigned char -> std::byte migration.
Supported format specs:
{} or {:x} — lowercase hex (default)
{:z} — hex with leading zero bytes stripped
{:a} — base32z
{:b} — base64 (padded)
{:B} — base64 (unpadded)
{:r} — raw bytes
{:W.Tx} — ellipsis truncation to W display chars with T trailing
The header also re-exports oxen::log::literals _format/_format_to UDLs
into session::literals for convenient use alongside the formatter.
Post-rebase cleanup and API improvements following the std::byte
refactor:
Crypto API improvements:
- Add ed25519::x25519_keypair() for combined Ed25519→X25519 key pair
derivation; remove detail::x_keys helper
- Add write-to-output ed25519::sk_to_x25519(out, seed) overload
- Make ed25519::sk_to_private variadic to forward to all sk_to_x25519
overloads
- Move PrivKeySpan runtime-size constructor body to .cpp (removes fmt
dependency from public header)
- Add encryption::BOX_*/SECRETBOX_* named constants alongside existing
XCHACHA20_* constants
- Add hash::ARGON2_* named constants for pwhash parameters
Namespace and naming:
- Rename session::encrypt namespace to session::encryption
- Rename decrypt_group_message parameter from
decrypt_ed25519_privkey_list to group_enc_keys (these are symmetric
keys, not Ed25519 keys)
- Change group_enc_keys type to span<span<const byte, 32>> (fixed inner
extent); simplify keys.cpp caller from singleton-per-key loop to
single call with all keys
Formatting and string handling:
- Replace all std::to_string usage with fmt _format (locale-independent)
- Replace "prefix" + to_hex() concatenation with "prefix{:x}"_format()
using new byte span formatter
- Rename formattable.hpp to format.hpp; re-export _format/_format_to
into session::literals
- Add fmt::range_format_kind disable for byte spans (prevents ambiguity
with fmt/ranges.h)
- Add Globals::session_id_hex() cached hex string
- Replace lazy-convertible to_hex calls in log statements with direct
byte span formatting
- Delete unused/buggy SessionID struct and fields.cpp
Code organization:
- Move wrap_exceptions, unbox, copy_c_str from public base.hpp to
internal config/internal.hpp
- Move hash::update_all into detail namespace (no external callers)
- Delete sodium_array (replaced by sodium_vector) and sodium_ptr
(unused)
- Remove cleared_uc32/cleared_uc64 aliases (no longer used)
- Audit and fix PrivKeySpan constructions outside try blocks in C API
wrappers (pro_backend.cpp)
Raw libsodium elimination (non-wrapper files):
- Replace crypto_sign_ed25519_* calls with ed25519:: wrappers in
user_groups, globals, pro, key_types, blinding, multi_encrypt
- Replace crypto_scalarmult/crypto_box calls with x25519::/encryption::
wrappers
- Replace all raw libsodium constants with named wrapper constants
- Remove vestigial sodium includes from protos.cpp, keys.cpp,
user_profile.cpp, session_encrypt.cpp, and others
Fixed-extent span conversions:
- Config key APIs (add_key, remove_key, replace_keys, get_keys, has_key,
key(), group_keys) now use span<const byte, 32>
- Remove runtime key size checks that are now enforced at compile time
Minor fixes:
- Fix URVO-breaking patterns (unnecessary local + return)
jagerman
force-pushed
the
pfs-pqc-pcs-dl
branch
from
737e1eb to
950feaf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch adds PFS+PQ (post-quantum) encrypted DMs, device linking, and a comprehensive std::byte refactor to libsession-util.
The core addition is a v2 DM encryption protocol using X-Wing key derivation (ML-KEM-768 + X25519), with a non-PFS fallback path for recipients that haven't published PFS keys yet. This includes the full encrypt/decrypt pipeline, a 2-byte key indicator prefix for efficient key lookup, and v2 message construction via Core::send_dm() with queued sends that wait for PFS key fetches. The session protocol layer was restructured: the destination-dispatch pattern was replaced with typed encode functions, the combined decode_envelope was split into decode_dm_envelope and decode_group_envelope (eliminating a type-confused design where Ed25519 private keys and symmetric group keys were passed through the same parameter), and large nested functions were broken up.
A new Core class manages persistent client state via SQLite, with CoreComponent-derived members (Globals, Devices, Pro) sharing a connection pool. Devices handles ML-KEM-768 and X25519 account key generation, rotation, and publication to swarm, as well as device link requests with emoji SAS verification. Core supports polling device and account pubkey namespaces, with callbacks for received messages, send status, PFS key fetch results, and link request approval. The predefined-seed mechanism allows restoring accounts from mnemonics, with Electrum word list support and checksum validation.
QUIC file server support was added via session-router, including a streaming file uploader that encrypts on-the-fly without buffering the entire file in memory. Live testnet integration tests exercise the full upload/download path against real infrastructure. The file transfer API supports progress callbacks, stall detection, and configurable timeouts.
The hashing infrastructure was overhauled: SHA3-256 and SHAKE256 hashers were added, blake2b_hasher was made into a streaming template class, and all hashing was made endian-safe. Integer values fed into hashes are now always written in little-endian encoding.
The cryptographic API was reorganized from a flat layout into algorithm-specific namespaces (session::ed25519, session::x25519, session::mlkem768) with headers under include/session/crypto/ and implementations in src/crypto/. Libsodium and mlkem_native headers are now confined to these wrapper .cpp files, keeping public headers dependency-free. A PrivKeySpan type provides safe, non-copyable Ed25519 key handling that accepts both 32-byte seeds and 64-byte libsodium keys with automatic expansion.
This also refactors all internal C++ APIs to use std::byte instead of unsigned char, replacing raw libsodium calls in non-wrapper source files with the new C++ wrappers taking byte spans.
C API wrappers were substantially cleaned up: snprintf_clamped was replaced with format_c_str for compile-time-checked format strings, and internal helpers (wrap_exceptions, unbox, copy_c_str) were moved from the public base.hpp header to internal-only headers. Various code was reorganized to be simpler and clearer.
A generic fmt::formatter for byte spans (session/format.hpp) supports hex, base32z, base64, raw, and ellipsis-truncated output, with _format and _format_to literals re-exported into session::literals. All std::to_string usage was eliminated (it is locale-dependent) along with string concatenation patterns for error messages. Config key management APIs now use fixed-extent span<const byte, 32>.
The dependency system was overhauled: internal dependencies were refactored to use session-deps, libsodium-internal was dropped in favor of a bundled tweetnacl implementation for X→Ed25519 point conversion; the new dependency system makes external dependencies significantly easier to update and maintain, and is designed to be shared across projects and submodules that have many of the same dependencies.