v0.5

Improvements:

• Per-transaction compression ratio variability is now active
  • CompressionReductionMethod::get_reduced_variable_global_size() and
    get_reduced_variable_local_size() now accept an optional transaction_id
    parameter and call get_effective_ratio(transaction_id) to compute the
    actual ratio, which may be perturbed around the nominal value when
    ratio_variability > 0
  • Engine::put() passes the current transaction id to both calls, so the
    simulated transfer size varies realistically from transaction to transaction
    when ratio_variability is set
  • The same transaction_id always produces the same effective ratio
    (deterministic hash of variable name and transaction id), making
    simulations reproducible
  • ReductionMethod base-class virtuals and all overrides (Compression,
    Decimation) updated accordingly; Decimation ignores the parameter as its
    reduced sizes are transaction-independent
  • Python bindings updated: get_reduced_variable_global_size() and
    get_reduced_variable_local_size() accept an optional transaction_id
    keyword argument (default 0)
• Transaction cancellation support
  • New Engine::cancel_transaction(transaction_id) allows an external actor
    to cancel a specific in-flight transaction, unblocking both its
    publishers and subscribers
  • Publishers and subscribers receive a TransactionCanceledException;
    catching it allows them to survive the cancellation and continue to
    subsequent transactions
  • Works with both File and Staging engines across any number of publishers
    and subscribers
• Memory efficiency: progressive eviction of metadata transaction entries
  • Once all subscribers have consumed a transaction, its entries are evicted
    from the in-memory Metadata::transaction_infos_ map
  • When metadata export is enabled and publishers and subscribers coexist
    (file streaming), evicted entries are progressively flushed to
    per-variable temporary files; the final metadata file is assembled at
    pub_close() from those files and any remaining in-memory entries,
    preserving the existing format and transaction count
  • When the stream is opened by subscribers only after all publishers have
    closed (sequential scenario), memory-only eviction is performed (the
    metadata file has already been written by pub_close())
  • Memory footprint of the File engine now grows as O(N_pub) instead of
    O(N_pub × N_transactions) for long-running concurrent streaming workloads
• Improved test coverage
  • Comprehensive C++ and Python test suite for transaction cancellation,
    covering cancellation while waiting for a subscriber, while waiting for
    a publisher, and mid-transaction for both File and Staging engines
  • Tests for the 3-parameter form of DTL::add_stream
  • Tests for propagation of reduction parameters to subscribers for both
    compression and decimation methods

Bug Fixes:

• SZ compression model: the ratio formula incorrectly used -log10(accuracy)
  instead of accuracy^beta, inverting the accuracy-to-ratio relationship;
  constants recalibrated to give ratio ≈ 7 at accuracy = 1e-3 and ratio ≈ 2
  at accuracy = 1e-6
• CompressionReductionMethod did not register the subscriber-side variable
  during inquire_variable, preventing Engine::get() from simulating
  decompression costs
• DecimationReductionMethod did not propagate its parameterization to
  subscribers, preventing them from querying reduced variable sizes after
  inquire_variable
• Transaction cancellation in StagingEngine: CancelException and
  NetworkFailureException thrown during pub_close and sub_close were not
  caught when a cancellation was in progress, leaving in-flight activities
  in an inconsistent state
• StagingEngine::begin_sub_transaction did not decrement
  num_subscribers_starting_ when a transaction was canceled, causing an
  imbalance in subsequent synchronization
• CompressionReductionMethod::ParameterizedCompression stored a raw pointer
  to the associated Variable; replaced with the variable name string to
  eliminate potential dangling references

API Changes:

• New Engine method:
  • Engine::cancel_transaction(unsigned int transaction_id) cancels a
    specific transaction performed by this engine. This method must be
    called from an external actor not involved in the transaction. It raises
    a TransactionCanceledException that publishers and subscribers must catch
    to survive the cancellation and continue to subsequent transactions. If
    both sides have already moved past the given transaction ID, the call is
    a no-op to avoid accidentally canceling a subsequent transaction.
• New Stream helper methods:
  • Stream::get_engine_type() and Stream::get_transport_method() respectively
    return the enum value of the engine type and transport method for a
    Stream.
  • As a consequence, in the Python bindings, the Stream.engine_type and
    Stream.transport_method readonly properties that return the engine type
    and transport method as a string have been renamed to
    Stream.engine_type_str and Stream.transport_method_str; the existing
    properties now return the enum values.
  • Stream::get_reduction_method(name) retrieves a reduction method
    associated to the stream by name, returning std::nullopt if not found.
• New ReductionMethod virtual method:
  • ReductionMethod::propagate_for_subscriber(publisher_var, subscriber_var)
    copies the publisher-side parameterization to the subscriber variable;
    the default implementation is a no-op; both DecimationReductionMethod
    and CompressionReductionMethod provide overrides called by
    Stream::inquire_variable.
• Behavior modifications:
  • DTL::add_stream can now take an Engine::Type and a Transport::Method as
    optional parameters (both defaulting to Undefined if not specified).
  • Reduction methods can now be declared in the DTL configuration file via
    a "reduction_methods" array in the stream definition.

v0.4

Major improvements:

• Data Reduction framework
  • New abstract ReductionMethod interface for extensible data reduction strategies
  • Decimation method: spatial subsampling with per-dimension stride, optional interpolation (linear, quadratic, cubic), configurable computational cost, and support for both publisher-side and subscriber-side application
  • Compression method: size reduction with preserved shape, supporting three compressor profiles (fixed ratio, SZ-inspired, and ZFP-inspired models), separate compression and decompression costs, and per-transaction ratio variability
  • Publisher-side reduction state is propagated to subscribers through inquire_variable, enabling detection and prevention of conflicting double reductions
• Reduction operations work with both File and Staging engines
• New documentation pages for the reduction feature
• Improved test coverage

API Changes:

• New Stream method:
  • Stream::define_reduction_method(name) creates a named reduction method ("decimation" or "compression") for the stream
• New Variable methods and properties:
  • Variable::set_reduction_operation(method, parameters) applies a reduction with key-value parameters to a variable
  • Variable::is_reduced(), is_reduced_by_publisher(), is_reduced_by_subscriber() query the reduction state
  • Variable::get_reduction_method() retrieves the applied ReductionMethod
• New ReductionMethod class with query methods:
  • get_reduced_variable_global_size(), get_reduced_variable_local_size()
  • get_reduced_variable_shape()
  • get_flop_amount_to_reduce_variable(), get_flop_amount_to_decompress_variable()
• Full Python bindings for all reduction operations
• Engine::put() now automatically simulates reduction cost and transports the reduced data size when the variable is reduced
• Engine::get() now automatically simulates decompression cost after receiving compressed data

v0.3

Major improvements:

• Enhanced code quality with comprehensive refactoring
  • Improved modern C++17 usage throughout the codebase
  • Reduced code complexity and technical debt
• Memory safety and robustness improvements
  • Fixed critical memory safety issue in DTL connection management
  • Fixed race condition in Engine creation
  • Plugged memory leaks
• Major refactoring with improved memory management, encapsulation, and move-only semantics
• Improved CI/CD infrastructure
  • Added Valgrind and sanitizer checks (AddressSanitizer, UndefinedBehaviorSanitizer)
  • Enhanced test coverage and validation
• Comprehensive proofread of the documentation

API Changes:

• DTL::get_stream_by_name_or_null() renamed to DTL::get_stream_by_name()
  Returns std::optional<std::shared_ptr> instead of raw pointer
  Python: dtl.get_stream_by_name() now returns None if stream not found
• Metadata management transferred from Engine to Stream
  Stream::get_metadata_file_name() replaces Engine::get_metadata_file_name()
  Python: stream.metadata_file_name replaces engine.metadata_file_name
• Stream method chaining improvements
  set_engine_type() and set_transport_method() now return Stream& instead of Stream*
  Enables cleaner fluent interface: stream.set_engine_type(...).set_transport_method(...)
• DTL::create() now accepts std::string_view and has empty default parameter
• String parameters changed to std::string_view for better performance:
  • DTL::add_stream(), DTL::get_stream_by_name()
  • Stream and Variable constructors and methods
• get_engine_type_str() and get_transport_method_str() now return std::optional<const char*>

v0.2

• New comprehensive documentation
• Full python bindings
• Improved code coverage
• Reduced technical debt
• Fixed deadlocks in File engine
• Update to SimGrid 4.1 and FSMod 0.4
• API:
  • Make metadata export configurable at stream level.
  • Have a simpler version of Engine::put() that automatically uses the local size of the Variable.

v0.1

This is the first release of the SimGrid Data Transport Layer Module (DTLMod), which provides a versatile simulated data transport layer for use in any SimGrid-based simulator.