OpenSSL Benchmark Wrapper

Description

This wrapper facilitates the automated execution of the OpenSSL speed benchmark. OpenSSL speed is a built-in benchmarking tool that measures the throughput and operations-per-second of various cryptographic algorithms, including symmetric ciphers, hash functions, and asymmetric key operations.

The wrapper provides:

• Automated OpenSSL speed benchmark execution across multiple cryptographic algorithms.
• Configurable thread counts with automatic CPU-aware interval generation.
• Configurable run duration per algorithm.
• Multi-iteration testing with averaged results.
• Result collection, processing, and verification.
• CSV and JSON output formats.
• System configuration metadata capture.
• Integration with test_tools framework.
• Optional Performance Co-Pilot (PCP) integration.

Command-Line Options

OpenSSL Options:
  --run_time <value>: Duration in seconds for each algorithm test. Defaults to 60.
  --threads <value>: Comma-separated list of thread counts to test with.
      If not set, automatically generates intervals based on CPU count (up to 8 intervals).
  --tests <value>: Space-separated list of OpenSSL algorithms to benchmark.
      Defaults to: chacha20-poly1305 sha256 sha512 chacha20 rsa4096 aes-128-gcm aes-256-gcm

General test_tools options:
  --home_parent <value>: Parent home directory. If not set, defaults to current working directory.
  --host_config <value>: Host configuration name, defaults to current hostname.
  --iterations <value>: Number of times to run the test, defaults to 1.
  --run_user: User that is actually running the test on the test system. Defaults to current user.
  --sys_type: Type of system working with (aws, azure, hostname). Defaults to hostname.
  --sysname: Name of the system running, used in determining config files. Defaults to hostname.
  --tuned_setting: Used in naming the results directory. For RHEL, defaults to current active tuned profile.
      For non-RHEL systems, defaults to 'none'.
  --use_pcp: Enable Performance Co-Pilot monitoring during test execution.
  --tools_git <value>: Git repo to retrieve the required tools from.
      Default: https://github.com/redhat-performance/test_tools-wrappers
  --usage: Display this usage message.

What the Script Does

The openssl.sh script performs the following workflow:

1. Environment Setup:

   • Clones the test_tools-wrappers repository if not present (default: ~/test_tools).
   • Sources error codes and general setup utilities.
   • Gathers system hardware information.

2. Package Installation:

   • Installs required dependencies via package_tool.
   • Dependencies are defined in openssl.json for different OS variants.

3. Thread Configuration:

   • If --threads is not specified, detects CPU count via nproc.
   • Generates thread intervals automatically using the test_tools generate_intervals utility.
   • Caps at 8 intervals for systems with 8 or more CPUs.

4. Test Execution:

   • Iterates through each configured thread count.
   • For each thread count, runs openssl speed against every algorithm in the test list.
   • Uses -evp flag for all algorithms except rsa4096 (which uses direct mode).
   • Runs each algorithm for the configured duration (default: 60 seconds per algorithm).
   • Records start and end timestamps for each algorithm run.
   • Skips thread/iteration combinations that already have output files.
   • Executes for specified number of iterations (--iterations).

5. Data Collection:

   • Captures raw output from each iteration in results_<iter>_iter_<threads>_threads.data files.
   • Extracts performance metrics:
     • For symmetric ciphers and hashes: throughput in operations per second.
     • For rsa4096: sign, verify, encrypt, and decrypt operations per second.
   • Optionally records PCP performance data during execution.

6. Result Processing:

   • Averages results across all iterations for each algorithm/thread combination.
   • Generates CSV file with configuration metadata and performance data.
   • Creates JSON output for verification.
   • Validates results against Pydantic schema (results_schema.py).

7. Verification:

   • Validates results against Pydantic schema ensuring:
     • All test names match expected enum values.
     • Thread count is a positive integer.
     • Operations per second is a positive integer.
     • Timestamps are valid datetime objects.
   • Uses csv_to_json and verify_results from test_tools.

8. Output:

   • Saves all raw output files, processed CSV/JSON, and system metadata.
   • Optionally saves PCP performance data.
   • Archives results to configured storage location via save_results.

Dependencies

General packages required: openssl

To run:

git clone <repo-url>
cd openssl-wrapper/openssl
./openssl.sh

The script will automatically detect your CPU configuration and run all default algorithms.

The OpenSSL Speed Benchmark

OpenSSL speed measures the cryptographic throughput of the system's OpenSSL installation. It tests how quickly the CPU can perform various cryptographic operations, which is critical for TLS/SSL performance, disk encryption, and secure communications.

Algorithms Tested

1. chacha20-poly1305: Authenticated encryption cipher combining ChaCha20 stream cipher with Poly1305 MAC. Widely used in TLS 1.3, especially on platforms without AES hardware acceleration.

2. sha256: SHA-2 family hash function producing 256-bit digests. Used extensively in TLS certificates, digital signatures, and data integrity verification.

3. sha512: SHA-2 family hash function producing 512-bit digests. Often faster than SHA-256 on 64-bit platforms due to native 64-bit operations.

4. chacha20: Stream cipher designed as a high-speed alternative to AES. Performs well on platforms without dedicated AES instructions.

5. rsa4096: RSA asymmetric key operations with 4096-bit keys. Tests four distinct operations:

   • sign: Private key signing operations per second.
   • verify: Public key verification operations per second.
   • encrypt: Public key encryption operations per second.
   • decrypt: Private key decryption operations per second.

6. aes-128-gcm: AES in Galois/Counter Mode with 128-bit keys. Authenticated encryption widely used in TLS, benefits heavily from AES-NI hardware instructions.

7. aes-256-gcm: AES in Galois/Counter Mode with 256-bit keys. Higher security level than AES-128, with slightly lower throughput.

Performance Metrics

Each algorithm reports:

1. ops_per_sec: Operations (or bytes) processed per second. For symmetric ciphers and hashes, this is throughput in bytes/sec. For RSA, this is operations/sec.

2. threads: The number of parallel threads used for the test.

Results are reported for each algorithm at each thread count, allowing analysis of both single-threaded performance and multi-threaded scaling.

Output Files

The results directory contains:

• results_openssl.csv: CSV file with all algorithm performance metrics, including columns: Test, threads, ops_per_sec, Start_Date, End_Date
• results_openssl.json: JSON file with validated results data
• results_<iter>iter<threads>_threads.data: Raw output files from each test iteration/thread combination
• test_results_report: Detailed test execution report
• meta_data*.yml: System metadata (CPU info, memory, kernel version)
• PCP data (if --use_pcp option used): Performance Co-Pilot monitoring data

Examples

Basic run with defaults

./openssl.sh

This runs with:

• All default algorithms (chacha20-poly1305, sha256, sha512, chacha20, rsa4096, aes-128-gcm, aes-256-gcm)
• 60-second run time per algorithm
• Automatically determined thread counts based on CPU count
• 1 iteration

Run with custom thread counts

./openssl.sh --threads 1,2,4,8

Tests each algorithm at 1, 2, 4, and 8 threads specifically.

Run specific algorithms only

./openssl.sh --tests "aes-128-gcm aes-256-gcm"

Benchmarks only AES-GCM algorithms, skipping all others.

Run with longer duration per algorithm

./openssl.sh --run_time 120

Runs each algorithm for 120 seconds instead of the default 60 seconds, producing more stable results.

Run multiple iterations

./openssl.sh --iterations 3

Runs the complete test suite 3 times and averages the results for consistency validation.

Run with PCP monitoring

./openssl.sh --use_pcp

Collects Performance Co-Pilot data during the run for detailed performance analysis.

Combination example

./openssl.sh --iterations 3 --run_time 120 --threads 1,4,8,16 --tests "sha256 aes-256-gcm" --use_pcp

Runs SHA-256 and AES-256-GCM for 120 seconds each at 1, 4, 8, and 16 threads, repeats 3 times, and collects PCP data.

How Thread Scaling Works

When --threads is not specified, the wrapper automatically generates thread counts:

1. Detects the number of CPUs via nproc.
2. Determines the number of intervals (capped at 8).
3. Uses generate_intervals to create evenly spaced thread counts from 1 up to the total CPU count.

For example, on a 32-core system, the generated thread counts might be: 1 4 8 12 16 20 24 28 32.

This allows measuring how each algorithm scales with increasing parallelism on the specific hardware.

How Result Averaging Works

When running multiple iterations (--iterations > 1):

1. Each iteration produces a complete set of results for all algorithms at all thread counts.
2. Raw results are saved in separate results_<iter>_iter_<threads>_threads.data files.
3. The wrapper extracts metrics from each iteration and sums them.
4. Final results are the arithmetic mean across all iterations.

This approach:

• Reduces impact of transient system effects.
• Provides more reliable performance measurements.
• Helps identify result variance across runs.

Return Codes

The script uses standardized error codes from test_tools error_codes:

• 0 (E_SUCCESS): Success
• 101: Git clone failure (test_tools)
• E_GENERAL: General execution errors
• E_USAGE: Invalid usage/arguments

Notes

Supported Platforms

• Linux: x86_64 and aarch64 architectures
• OS Support: RHEL (dependencies defined in openssl.json)

EVP vs Direct Mode

• Most algorithms are tested using the -evp flag, which routes through the OpenSSL EVP (Envelope) API. This is the recommended interface and engages hardware acceleration (e.g., AES-NI).
• RSA4096 is tested in direct mode (without -evp) as it uses the asymmetric key API, which reports sign/verify/encrypt/decrypt metrics separately.

Performance Considerations

• AES-NI: Systems with AES-NI hardware instructions will show significantly higher AES-GCM throughput. Check with grep aes /proc/cpuinfo.
• ChaCha20: Performs well on platforms without AES hardware acceleration, making it a good comparison point.
• RSA4096: Asymmetric operations are orders of magnitude slower than symmetric operations. Sign and decrypt (private key operations) are much slower than verify and encrypt (public key operations).
• Run multiple iterations (--iterations 3 or higher) for reliable results.
• Ensure system is idle during testing for best consistency.

Performance Tips

• Run multiple iterations (--iterations 3+) to verify consistency.
• Ensure system is idle (no other workloads) for best results.
• Disable CPU frequency scaling (use performance governor) for reproducible results.
• Consider the active tuned profile on RHEL systems.
• Use longer run times (--run_time 120) for more stable per-algorithm measurements.
• PCP monitoring (--use_pcp) adds minimal overhead but provides detailed metrics.

Troubleshooting

• If openssl is not found, ensure the openssl package is installed.
• If results seem inconsistent, run more iterations and check system load during testing.
• Use --use_pcp to collect detailed performance counters for analysis.
• Check results_*_iter_*_threads.data files for detailed error messages from openssl speed.
• Verify hardware acceleration support: openssl engine and grep -E 'aes|avx' /proc/cpuinfo.

References

• OpenSSL Documentation: https://www.openssl.org/docs/
• OpenSSL Speed Manual: https://www.openssl.org/docs/man3.0/man1/openssl-speed.html
• test_tools Framework: https://github.com/redhat-performance/test_tools-wrappers