From 71c44d4ef69329699f990bcfe0bab378e3d35e54 Mon Sep 17 00:00:00 2001 From: "Sherman, Srikanth" Date: Tue, 4 Aug 2026 20:28:59 -0700 Subject: [PATCH 1/7] Add NGINX with Intel QAT optimization guide Add the NGINX + Intel QAT workload guide covering hardware and software prerequisites, async-mode-nginx build and configuration, and CPS benchmarking results. Includes the with/without QAT configuration pair used for the comparison and the openssl s_time driver script for the handshake test. --- software/nginx/README.md | 265 ++++++++++++++++++++++++++ software/nginx/connection_test.sh | 78 ++++++++ software/nginx/image.png | Bin 0 -> 22892 bytes software/nginx/nginx_with_qat.conf | 62 ++++++ software/nginx/nginx_without_qat.conf | 40 ++++ 5 files changed, 445 insertions(+) create mode 100644 software/nginx/README.md create mode 100755 software/nginx/connection_test.sh create mode 100644 software/nginx/image.png create mode 100644 software/nginx/nginx_with_qat.conf create mode 100644 software/nginx/nginx_without_qat.conf diff --git a/software/nginx/README.md b/software/nginx/README.md new file mode 100644 index 0000000..51ea885 --- /dev/null +++ b/software/nginx/README.md @@ -0,0 +1,265 @@ +# NGINX with Intel® QuickAssist Technology (Intel® QAT) Optimization Guide +## Table of Contents + +- [Overview](#overview) +- [QAT Hardware Requirement](#qat-hardware-requirement) +- [QAT Software Requirement and Prerequisites](#qat-software-requirement-and-prerequisites) + - [Enabling the Required QAT Services](#enabling-the-required-qat-services) +- [async-mode-nginx Configuration](#async-mode-nginx-configuration) +- [Building and configuring async-mode-nginx](#building-and-configuring-async-mode-nginx) + - [Generating the Server Certificate](#generating-the-server-certificate) + - [Validating the Configuration](#validating-the-configuration) +- [Files in This Directory](#files-in-this-directory) +- [Benchmarking](#benchmarking) + - [Core Allocation and `worker_processes`](#core-allocation-and-worker_processes) +- [Results](#results) +- [Details](#details) +- [References](#references) + +## Overview + +Compression and cryptography take up a significant portion of resources in the data center. Hardware acceleration like Intel® QuickAssist Technology (Intel® QAT) can be used to offload the compression and encryption portions of a workload. Offloading these operations will free up CPU cores to do other work and will improve compression and cryptography performance. NGINX is the world's most popular webserver. It is free and open source software, distributed under the terms of a simplified 2-clause BSD-like license. The "Async Mode for NGINX" adds asynchronous capabilities to NGINX using the OpenSSL Async Infrastructure. + + +## QAT Hardware Requirement + +At least one Intel® QAT engine is required and the individual engine might need to be updated in the BIOS. The following steps should be performed to be ready to use the QAT device(s). + +1. Check for QAT device availability. This can be verified by running the following command: + +``` +echo `(lspci -d 8086:4940 && lspci -d 8086:4941 && lspci -d 8086:4942 && lspci -d 8086:4943 && lspci -d 8086:4944 && lspci -d 8086:4945 && lspci -d 8086:4946 && lspci -d 8086:4947) | wc -l` supported devices found. +``` + +The command reports how many supported devices were found. At least one is required. On the system used for this benchmarking, the output was: + +``` +8 supported devices found. +``` + +2. Verify that the QAT firmware is already loaded by using the following command: + +``` +ls /lib/firmware/{qat_4xxx,qat_402xx,qat_420xx}.bin* 2>/dev/null +ls /lib/firmware/{qat_4xxx,qat_402xx,qat_420xx}_mmp.bin* 2>/dev/null +``` + +The output of the above command should include 2 firmware files. Note that this can vary depending on the exact QAT device on your hardware. + +``` + /lib/firmware/qat_402xx.bin + /lib/firmware/qat_402xx_mmp.bin +``` + +If the firmware is not already available, it can be downloaded from the Linux kernel repository: +https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/intel/qat + +``` +cd ~ +wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_4xxx.bin +wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_4xxx_mmp.bin +wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_402xx.bin +wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_402xx_mmp.bin +wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_420xx.bin +wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_420xx_mmp.bin +sudo cp qat_4xxx*.bin qat_402xx*.bin qat_420xx*.bin /lib/firmware +rm qat_4xxx*.bin qat_402xx*.bin qat_420xx*.bin +``` + +After firmware is updated, the initramfs must be updated. This differs based on the Linux distribution. + +3. Verify that the kernel drivers are loaded using the following command. + +``` +lsmod | grep qat +``` + +The output should be similar to the following: + +``` +qat_4xxx 16384 0 +intel_qat 172032 1 qat_4xxx +``` + +If the kernel modules are not found, they can be installed using: + +``` +sudo modprobe intel_qat +sudo modprobe qat_4xxx +``` + +If the kernel modules could not be installed, it might be needed to either install them through a kernel configuration or to install them with the distribution's package manager. + +## QAT Software Requirement and Prerequisites + +The QAT driver is available either "in-tree" as part of a release kernel or can be built outside of the release. This document assumes the use of the in-tree driver that is already available with kernel after version 5.19. The distribution used for this benchmarking was Ubuntu 24.04 with the in-tree driver. + +QATLib provides user space libraries that allow QAT device access and expose APIs for use by higher level applications. The QATLib driver can be installed using your distribution's package manager. For Ubuntu 24.04: + +``` +sudo -E apt install -y libqat4 libqat-dev qatlib-service qatlib-examples libusdm-dev +``` + +QATzip is a user-space library built on top of the Intel® QuickAssist Technology (QAT) user-space library. It provides extended compression and decompression capabilities by offloading these operations to Intel® QAT Accelerators. + +``` +sudo -E apt install -y qatzip libqatzip3 +``` + +Depending on the use case, the user can configure the number of QAT engines to use with the workload. In "Managed Mode", the [QATLib](https://intel.github.io/quickassist/qatlib/index.html) library can be used to restrict the workload to a specific number of engines. + +Please note that "intel_iommu=on" will be required as a kernel parameter. + +### Enabling the Required QAT Services + +Each QAT device is configured by a `/etc/4xxx_dev*.conf` file, and the `ServicesEnabled` setting in the `[GENERAL]` section controls which acceleration services that device exposes. This setting must include the services your workload actually uses: + +| ServicesEnabled | Services available | +| --- | --- | +| `dc` | Compression/decompression only | +| `sym` | Symmetric crypto only | +| `asym` | Asymmetric crypto (public key) only | +| `sym;dc` | Symmetric crypto and compression | +| `asym;dc` | Asymmetric crypto and compression | + +This matters because the two optimizations in this guide use different services. The qatzip module (`ngx_http_qatzip_filter_module`) needs `dc`, while QATEngine handling TLS handshakes (`ngx_ssl_engine_qat_module`) needs the crypto services. A device left at the compression-only default will not accelerate TLS, and the CPS results below cannot be reproduced on it. + +Check the current setting: + +``` +grep -H ServicesEnabled /etc/4xxx_dev*.conf +``` + +To use both compression and TLS acceleration, set the following in each device's `[GENERAL]` section: + +``` +ServicesEnabled = asym;dc +``` + +Then restart the service and confirm the devices come back up: + +``` +sudo systemctl restart qat.service +sudo systemctl status qat.service +``` + +Note that the available `ServicesEnabled` combinations vary by QAT generation, and not all services can be enabled on a single device simultaneously. See the [QATLib Users Guide](https://intel.github.io/quickassist/qatlib/index.html) for the combinations supported by your hardware. + +## async-mode-nginx Configuration + +This optimization was tested with the following software versions: + +async_mode_nginx v1.0.0 +nginx 1.26.2 +OpenSSL 3.0.13 +QATEngine 2.0.0-1~noble1 + +QATEngine is the OpenSSL engine that `nginx_with_qat.conf` selects via `use_engine qatengine`. On Ubuntu 24.04 it can be installed with: + +``` +sudo -E apt install -y qatengine +``` + + +## Building and configuring async-mode-nginx + +[async-mode-nginx](https://github.com/intel/asynch_mode_nginx) can be built with: + +``` +./configure \ + --prefix=$NGINX_INSTALL_DIR \ + --with-http_ssl_module \ + --add-dynamic-module=modules/nginx_qatzip_module \ + --add-dynamic-module=modules/nginx_qat_module/ \ + --with-cc-opt="-DNGX_SECURE_MEM -I$OPENSSL_LIB/include -I$ICP_ROOT/quickassist/include -I$ICP_ROOT/quickassist/include/dc -I$QZ_ROOT/include -Wno-error=deprecated-declarations" \ + --with-ld-opt="-Wl,-rpath=$OPENSSL_LIB/lib64 -L$OPENSSL_LIB/lib64 -L$QZ_ROOT/src -lqatzip -lz" + +make +make install +``` + +### Generating the Server Certificate + +Both configuration files in this directory expect a certificate and key at the paths below. These are not created by the build, so generate them before starting the server. The results in this guide used a 2048-bit RSA key (RSA2K): + +``` +sudo mkdir -p /usr/local/nginx_qat_module/certs +sudo openssl req -x509 -newkey rsa:2048 -nodes -days 365 \ + -keyout /usr/local/nginx_qat_module/certs/server.key \ + -out /usr/local/nginx_qat_module/certs/server.crt \ + -subj "/CN=localhost" +sudo chmod 600 /usr/local/nginx_qat_module/certs/server.key +``` + +This produces a self-signed certificate, which is appropriate for benchmarking but not for production use. + +### Validating the Configuration + +Before running a benchmark, confirm that the configuration parses and that any dynamic modules it loads are present: + +``` +$NGINX_INSTALL_DIR/sbin/nginx -t -c /path/to/nginx_with_qat.conf +``` + +A successful check reports: + +``` +nginx: configuration file /path/to/nginx_with_qat.conf test is successful +``` + +This step catches missing module paths, unreadable certificates, and syntax errors before they show up as a failed test run. + +## Files in This Directory + +| File | Purpose | +| --- | --- | +| `nginx_with_qat.conf` | async-mode-nginx configuration with the QAT modules loaded and the QAT engine enabled. | +| `nginx_without_qat.conf` | Baseline configuration with the QAT modules commented out. Note that this still uses the `asynch` listen parameter, so it must be run with the async-mode-nginx binary rather than stock nginx. | +| `connection_test.sh` | Drives the Connections Per Second (CPS) handshake test using `openssl s_time`. | +| `image.png` | CPS results chart. | + +## Benchmarking + +CPS is measured with `connection_test.sh`, which spawns 200 concurrent `openssl s_time` clients against the server for 10 seconds each and sums the per-client connection rates: + +``` +./connection_test.sh +``` + +To print the commands without running them: + +``` +./connection_test.sh --emulation +``` + +The client count, duration, port, and cipher are set in the USER INPUT block at the top of the script. Note that the script's default cipher (`AES128-SHA`) and the `ssl_protocols TLSv1.2` setting in both configuration files do not match the TLS 1.3 / ECDHE-X25519-RSA2K configuration shown in the results chart below; adjust both to reproduce those specific numbers. + +### Core Allocation and `worker_processes` + +Both configuration files set `worker_processes 48`, which is deliberately a fraction of the cores available on the test system rather than all of them. + +This reflects the scenario the guide is intended to demonstrate. In a real deployment, a web tier rarely has an entire high-core-count server to itself — it shares the machine with application, caching, or database workloads. The question that matters is therefore not "what peak CPS can this server reach with every core dedicated to NGINX," but "how much TLS throughput can be delivered from a modest slice of the machine, leaving the rest for other work." + +Offloading handshake cryptography to the QAT devices is what makes that slice go further. Because the asymmetric crypto moves off the cores and onto dedicated accelerators, the same 48 workers sustain substantially higher connection rates than they do without QAT — so the cores that remain free are genuinely available to other tenants rather than being consumed by TLS overhead. + +## Results + +![NGINX TLS handshake CPS, C3 SPR vs C4D Turin vs C4 GNR with and without QAT](image.png) + +Intel® QAT is only exposed on bare-metal cloud instances, so this comparison is run there rather than on virtualized shapes. The two C4 GNR bars are the same bare-metal Intel Xeon 6980P system described under [Details](#details), both running `worker_processes 48`, with the QAT modules and QAT engine as the only variable between them — the "without QAT" bar corresponds to `nginx_without_qat.conf` and the "with QAT" bar to `nginx_with_qat.conf`. The C4D Turin instance is likewise bare metal, so the cross-platform comparison holds the provisioning model constant as well. + +## Details + +NGINX on GNR 128c (Intel Xeon 6980P), bare metal: 1-node, 2x Intel(R) Xeon(R) 6980P, 128 cores, 500W TDP, HT On, Turbo On, NUMA 6, Total Memory 1536GB (24x64GB DDR5 6400 MT/s [6400 MT/s]), BIOS F23, microcode 0x10003f3, 2x I350 Gigabit Network Connection, 4 QAT engines, 1x14.3G SanDisk 3.2Gen1, 8x3.5T Samsung MZQL23T8HCL5-00A07, 1x7T Micron_7450_MTFDK8G1T9TFR, Ubuntu 24.04.3 LTS, 6.8.0-86-generic. Test by Intel as of Nov 18, 2025, async_mode_nginx version 1.0.0, nginx 1.26.2, OpenSSL 3.0.13, QATEngine 2.0.0 + +Results may vary. + +## References + +asynch_mode_nginx: https://github.com/intel/asynch_mode_nginx + +QATLib Users Guide: https://intel.github.io/quickassist/qatlib/index.html + +QATzip: https://github.com/intel/QATzip + +QATEngine: https://github.com/intel/QAT_Engine diff --git a/software/nginx/connection_test.sh b/software/nginx/connection_test.sh new file mode 100755 index 0000000..dc619be --- /dev/null +++ b/software/nginx/connection_test.sh @@ -0,0 +1,78 @@ +#!/bin/bash +###################################### +############# USER INPUT ############# +###################################### +ip_address="$1" +_time=10 +clients=200 +port=443 +cipher=AES128-SHA +###################################### +############# USER INPUT ############# +###################################### + +helpAndError () { + echo "This script is used to run the ConnectionsPerSecond(CPS) testing HTTPS." + echo "To use this script: ./connection_test.sh " + echo "To do a dry-run, use the emulation flag:" + echo "./connection_test.sh --emulation" + exit 0 +} + +# Check for h flag or no command line args +if [[ -z $ip_address || $@ == *-h* || $@ == *--help* ]]; then + helpAndError +fi + +# Check for emulation flag +if [[ $@ == **emulation** ]] +then + emulation=1 + +fi + +# cmd1 is the first part of the commandline and cmd2 is the second part +# The total commandline will be cmd1 + $ip_address:$port + cmd2 +cmd1="openssl s_time -connect" +cmd2="-new -cipher $cipher -time $_time" + +# Print out variables to check +printf " IP Addresses: $ip_address\n" +printf " Time: $_time\n" +printf " Clients: $clients\n" +printf " Port: $port\n" +printf " Cipher: $cipher\n" + +# Remove previous .test files +rm -rf ./.test_* + +# Get starttime +starttime=$(date +\%s) + +# Kick off the tests after checking for emulation +if [[ $emulation -eq 1 ]] +then + for (( i = 0; i < ${clients}; i++ )); do + printf "$cmd1 $ip_address:$(($port)) $cmd2 > .test_$(($port))_$i &\n" + done + exit 0 +else + for (( i = 0; i < ${clients}; i++ )); do + $cmd1 $ip_address:$(($port)) $cmd2 > .test_$(($port))_$i & + done +fi + +waitstarttime=$(date +%s) + +# wait until all processes complete +# The bracketed first character keeps grep from matching its own command line. +while [ $(ps -ef | grep -c "[o]penssl s_time") != 0 ]; +do + sleep 1 +done + +sumTotal=$(cat ./.test_$(($port))* | awk '(/^[0-9]* connections in [0-9]* real/){ total += $1/$4 } END {print total}') +printf "Connections per second: $sumTotal CPS\n" +printf "Finished in %d seconds (%d seconds waiting for procs to start)\n" $(($(date +%s) - $starttime)) $(($waitstarttime - $starttime)) +rm -rf ./.test_* + \ No newline at end of file diff --git a/software/nginx/image.png b/software/nginx/image.png new file mode 100644 index 0000000000000000000000000000000000000000..87d92b55f11710e3ba93fddd94a4a8b4fb65e572 GIT binary patch literal 22892 zcmeIa2UL^W)-G&Eq=}$(5S1Q4rAt##QA1IBs8RybdoNLGB26g)2}RU^)IjK6X`!Q` zClnzDqyz{Z0^~;Bd)$5Y;oSTEW1Rp0#y`Hp7(nvAS!K@o%xBIu^R5;7P*aWW4EvcQ zM~=|lzjs^v$dThf$}jEdWTp-kh+#rE%-Xk&-x?J&Ti+ z_l^(Mb?=P)`WYP^C6mb(78W27sJFK_J3BixG_<+7IWsdeFfg#VxY*s@eSUsECME`h z!Pwi|L!r>>>S{vJH3I_!N>7Z8jEjqlc6N4`FJB%XACHWTEH5wr^5siPN{YX~zoevO zPEL-Rnwn~D*V&(w^3>MUfzKGl9i_aXcD)DkIC6xA@#pW+PX0CJBS#Da@87uW|x zr}lw87%rF$xLP4K5+5&)l*phZla(jR1D4_+Ks_pA9OpqI*HoBKda}^bJtyxkllL!& zZujjiLuBusCK~K5XF^W#wm|NH&NWei@1N#L4c{7hlbea2s%}W^=LV z-tawXWn>1y3IUV5WyM?v96_-ntfj1Kto^II*rnFeGHz5YNpfsa_yVlfjnhB|yE5m` z+5E|dQ^xLjzkGTIEI))$(J&r^TT=C|U^Kv0VFwsd{Orzc&_dDGn&8@EzcHsT`M7o@ zK)Tkg4x>;Hs{;l7Q2Vm$y8l^vk||21+I%7gH#pBZxju?1ws3YIAZ~ z_YBn)fQ^N-L9rL|2 zD^)(`0!^O3H|}WXxV%g{wD!4zsHrAjMT$J;}oWOj|K- z{;E@!0HrecOQ!RwN2Ee{P{g5`61g!i+gP!#dGC6zC6~4w zpC%Zw_h5j3=ds!AS~ruRvYyHtCYusf9EhRjV$;ddSbVdr*mB{OCi>-jga%At9ZvzZ$FU_gIKdc#X-wqTe}3{^m=&SSTt_+GS<1) zbFnJ=wW4OmM|TBarCKb+UsJ4$YP0pI*I*p5fIH`N4_hIZ`?y_E4K!?csdfoQsym(U z%;4r!%khHckM4)6SJ3lL^!DC8X!ivB%{e-~>QP7R^c*N#yr|gMc=dJ?4rIi|%(&-mjE8?=^zz~uw!bz(ABA+&cObuAyw280DC)E2u`2%|Nt z_s`a-qJ`YbsvDz_BTnm!(tDt~2T=F9*1=l2Cr}JPYNK%}WZB1RZpvu(alu-{>)}bP zp*zF0#WHvwPhGW7*(_KK&i~SRD5$61vOTCY!CO$Cj z8=4Lo;0EoJ`gRR)8fhAPRSva}3l_?H(`@XpJTyg}IU(OUqmRy6Kp#YCJ!lu+aJa2E zFI4@2SnkoPXL2-OzbA3nPx=vy7SrV6nYBg>2<$V^) zZWZ!~uXmle2VEMjZD1ejarga>wVk`^p905^_iS7>nQYZTgb{9BX!btuQ@uqI-LlHu zto4*9mpyvF4e+?Lbtz>Z$lUB=`!V7qV1E&WMDn2Y`c}5U#d=%hiC>vX+ z4W0JWGD+CnyE0g(1e{vW(-o9YEmHHn>zKPde@a0cvXD_Ax{SXia6A3uGn)?0AOuE- z?2!x}!LR{6<~bNvz4a%u8hNdW_n^~tHNq2v69dOhk<4O`oKQsP^Djt0_MB=RqXrF+ z+f&`065KO{MlUb~!6t0xS{RYf(9KqL7FA{wMT-Ez!7$nDMYbLHF|ytbiLqK153ubp zWbaV6jBTl#$CW6S=_DkxeCBj&8B0K{jZm6d5}g-Bs_BulZ1sqAiL4O>f#`Je&kdqt zl4g=hYuA1y8H4rfvK-MXpLq}d;m^6;93&|eH4DyN5V`No2ejz)9_9()r!Qq@k6~Z( zQaED^dTEfikYl>z>RUbaIAH_b3`l}$$AWXa3)gX`c$`kRb~VQp@EpI7>Id#Kxb|BO z9Ec?NmzeW1LtY6nI1o~27May_VZP#1Iet~uW5lE2ph8G!-&b~uD45=>J?Y~&dMNQj7{oWn`fDgDyHp?q)5fFENN;;BiK3 zQv7Amx4g(y=hZ9p_hAV>`J3ghJ|%in%V%(914M^%`)0xsNdgEc-$I>eq)^h7FVwNL zpnn%@P-2U4mvIg^)d^RdUk-xBqrLV^l-m~g?Pb&94GrlffJi0ag{GwQ8{<_SImWZ# zxteOFWJFsMp!L-I#^qPmx#pGHJTU4TH8($;mVs5cS9NIG3T@eO_o4HY&0mh4Fi3Gv zmIn63OV!&!#YZ^Ln~4IwX*4k}qc8*UfKmKjRo0Rd&4vMslaATw)T5n1Lr9@SS+KdY zL&rEbiZY8gaFRPv_uCx!{@C)a=7;nRdBaPTgT z%KevIeP|7mkk^q_z$oF?c#dpWQN%U(;tv10ThP@Pm^hZk80rp#%e72c)>xcc$X0H@ zShuLD*@J*ImYEG?Sr1i9iB(t8$xxU?99YOZX!KLW7rTR_m;|?O`K+5d3MB|R^LJg2 zZN!=2s)m^HQ`r1sY+IzTTB&Ey#uRSP0?o3@w`|_`uXts{>1WgG zw{zIoFwa+;pL>E8yXge-JVBkE?jSDBvl&LvyoG>=j)AOJ%FZv}uU4Fyopv(_005i4 zMgzT@eRM|@uWyXi0Pd$qSPMaH%L~Rl&W7Ou^<|S@&jT-BxPVs@SCNrk6u%s$YWMvr z5Lhc@A!A0M_>9S;;)Jd;yLeG`y9oKIiEw^kwFW>r*{>gzr!3lFK4|`|HFGK^f1Tq+ z;L&dtoU`UAj}LW?QI*Qfj*YNWFmujP)WC&-*CnapwN>78IUc#DH&Q;HZPW^h+*omA z66^d{&!i;^tkD3#4BU@b0)%L`aHD99qHo3N9aI99aw?57l4A|Aqhh4)sdqtaEY2q> z*nzFLYucvb!ko?d2AcDG!9g6siEoz1L>wN*h2ap=%hN6=9U@yMnP#vz(IYjJFG_yE zU%Qv&6hXN%7L~>(!^=&-d-ZGqFrq%S_Vd14#wYh`HP$|-g8$J z$Kon$dwc@wPK#&o5!nY$nW~2)*To(}#g2gIOy2@bk94AIZ-Q!$B*J}eEt#HWdIzAM z3PD~8W$27Qt0V|i*?#-4FPy{UN+GfpbHUbpHsQ$I)$+bgy*hykfr;=GhGiuno@I0& zaVIl-ufc;=2^06UzfZMaD|?OM5ot?rNmE80uU6yNONj2*GJ;J8PYj^M;pVYJJWCao z$RQTj?`mHqm6er#_^H2Lah`$_W{O9_2J;CEU8~^pX(6+$fb- zYK68~XWST$fiwlr@QUm&IOq4MxmjklZZ(AQ8^~5F?(5t)ulLe%tEBBZ4Yc~|@#pPK z0~2%<(T?uEEj-iEbfBbc(UehZS-ZWkT?AcMuLS$MYHc+5H1{X(c)oF?n_j<;@a1VYR2?w^MFMrr3cGFSpYuQmvPm>c!Mh-jse}-ZS^k)K}kKT#5lBV zWVftBlCUzfpjN;Q;Et%?epKwLM68zWoOhkk@aPOxhtuPnS2EwErEMt#j7Q5*As$c8 z;GR}tI5tm|`-C79KM5&)U>{K`85P_R8xT$-IvK!Tivc~`@T1YTwjJyr;<73#j<#$4 zP)LXOtTyyk6lkuUOFM|#6RdHDLZ?L^G<~(+=78c(Wrl4Yq!>fd??0kHhbz9pnB9Jp zd1p{TPDByhv-xHG_5B36ULj%wtCM3r-s0H?*OOHSZkBBZI9~4Sig!!l3;Z6ub4e-Z zytzzG#ada&{yi9cUAn}lryFUlWmi`Dvc+iY^F*O#gp?XGQG{QqOJTT{G*aNXobMP* z8;1r*_Eag+t+0WRV~?#~XS(effSwzC{w|I#jU#%!pQ;YWkJrB0d-^cICK$54qx1#@ zU9#kQvAn0Q#wU~2GWEEyPWV_~lSe%eF-Vv&@bm2+?*G*#d5;bBlFMO#=U(uo;6+qz zjYf?AB5Hv<2xMc-%@nKV=2pMx)<(RA-(R>}nZ?y=V3wHugj1Q*Z*iMsWd@ozH#VCT za7%%oJ0tr7xZzCVHP@P(Ft971w}z(9EN@3&<^v|B!P`F*RH@(CA;-EnvF}V$;Xvl+3|WmH(H0s&mRqFR9Rv|Xf+1+b&CM;uawpSiNofs z5ukaoSmt?n?-;Asa=wD>QI`%hOS`r&gS-4YKk=3(cDjXL`nl@LjWNvewXS&m1^4P` z>s+&o_U1CF6L0z(UklVD_$q+Q`)KXQW*|&V?RHV?_5@3SPEg?VW(fdKq|$3pv^*n8 z(YZj}268U?sa0JF+wAAHSRz{QN7bIsW_&uog|`%y{j%wBUwwVvs6!!+G{Z7A2p!~g zyeZ$}uxex`V^ow^&UK+D;3>+@4q+!|UT-jy1TXWEQu&Gj8lXj57)led@Pm8gZ6Tna zIXIwqMHl)bTDdLE?=Ex^Z&s<{A6w6m6ELa$YWDNx1!nr^yg-Z3n4o+9<4@%59rdN_ zOC=uXo5spcUR&!#Pj%@LxTdF8a94rEHBpmt1?^=^gjvd(&MYuyKax$kR%>s<>=u_9 zHc^QuQri@MPXf_AI#U^JgRFqCnNYe^6a(Z*KX5Bwt7ltI@dMATz$fuTC!z2eLq1o- z;kwliA}vftnMhu19y-v z{w}_9el!4YqDJzZ)90?F`qH{F7{|v_Q%v&kID_CZK46yHf|(4)O#dZYOJ#q~V8gTU z?eS{}fK*UJ3E%1O$Ma?cPU!Ojlii2DXWku%+}+ZP4{3jpA=@+kxGmE3hPHev{I;>k zLW${*ItMe)HxsWoBVB3$rN#C4hut32?@+N-eyM@dvBoOb4#!NCX}CRkmD=#4SZWoW zJ$$m!c;+>H+x`CKvAE<3f?s1A$4cYT5Y~PZZI)lExky0*7;p0w$WhR8c z98scM*0oZ|<4FO1FwKZf?nEY&^XJm83O3`O?d+~qrQQ|qeoT%zWD*QA8cB{o951I! z=`C*(ohB^Yg{ju9l~+Kg?} zLf>IQV07H}(yGKgnf@1`cYsG3JhX}tPlg16iE$&>_O4Vy++Olc(ES%e?BPLJt*XM@ zYu4OeeDQuZ9*=ml@(nlm)nH_X2Zj>wd#7~s^Ec%U!-ADtw0)Ki%AL#HaMjc=Z}KtUadA51e{~6+mo_HplH{%BqRHN#H0at!g*y(b7kg^>z%-7Xx;_#YUi+5U~P=zxQMbJSfE* zHx*?*1OUBT>vFiG0#b4Alu%nAiQWeBQ%l^?ma|FAr2RzG1!5JsN}Wa zvFu$8?3jqv`2+x=nr}fuf!U(gSV5;9y8HWWnxozzQeCE?kza_eui)!qd6IWv@ReABD*b?} zfHyLo=UhF$OrGCy{SYPMb%v)89R#^0(~$F{NKzlm&;$P34(Uce7B6%%Q-4xPJH%JD zgzcEA7t?StmuIx@h(EsXyRwQIeGfQ_4e^=2Ln$^49pFIM?*zO8FZW?9oCbsDV+9id zcv#@o8&sDmRXkRo=Ir#Gdei#AACoM`livqyI*jn{Sk+8EAiXuUnVEb*BHxFGtkr`G z{p$f6YJsCxB4x#=4C|*g8hC>c zM_jaW23!LU<5JA@jf)01T|T8NcAm3D4Az>q1n3PnTW;~Dag3D@ZrW6;vlynrKs5Nv zoX1B`Vp#iId(CKSdn;wEA3ztOpGco4u96A>qbKp!)ku;~Wu@tX0nXP}{D)~Jf!A7S z-3Y@z2{E|^ZOWG*0Y1YQQA3X0 zNCHAvwQisTl=^z(MGZ{4_nIS@A}-+-kv(3#-RPCC=Qtyso^-?$tSQm17XPG85U6H2 zX*Q}guJW*gr@(BpF`(9bG92dL8Aq%8!MwsMw&U}4q$6DsOGcdYGyV5Xdu6?}|0dlre*MKWS@?!46B z*A3sL4o%7E>O{NKHs+6JcYn9Z`_(rW`SdlqcB zQDDjB+Zc$HX==5-|3c>Ze{n5(c%b~be2w&5h1fp75sSPsLv0iO?El{{dc&;kOkKJ1pP*42F*`v^n$^5>W%|j7tL6I_1M*^{V6q#HJ&wP^B1V5 zs~_92e#DtPfiA}D%)?8w4s=-a-iL0}T$P|5aHGJ!*4H|@sJ2aqmxDz!93Z)&Bhe6j zZhGf;tJ_e`cNJ=NCsNo>&C9Z6!og4Ap{}QX2{m2%A>#(|((lg&LPLESfAyHicTMX* zFaF(15W52Ga$`CuLb-8PN|&*rB#X$uxX73Lw`E?#onl+PIm0b%EsSECiv>Zse*o~r zFI!zgmSO2MgfHOS4WpyJtcUXF?fh;o_Gfd8C#7Xv$<1x-A1%0!S-tM~2mBr@En^*S zwr7ceTB!f#K;J(artrx#y?j)1K{#6IXNy}XWfcXw-*`2G4&jVI=fvrMzcqwv`aH>bj|%s&TPp?b9FR=>!Dm@)i1XELH(2jEGc_L&frwjLozjG65@XfMQD3!>QnyNm0p)>vHN4-jqA_O zLVknrkC+X7`YW~&K}O;9^1Nxk9bg-fo1tk7G`hup$Vwyi|<7{&4mLh7tuIL0fJ= zhJ)YsQkPL;=6gV7 zJ5-tvXh%t2TKpatTKqQuFAOX(vn0WLs%R{Q)9 z-v=4LYk~{`eZK4bsLNNK7XSP=%^;DI1)Ykwy=7jf(}r~?YI(h7I|rc}V*dmGzny>u z3Ri{N`F$2X#jb9aw+6#BaYa@rOd^KL%X;`;68Bk;%vZQJ0)xyM}+% z8pU&ZSpGInB7EohKgu7Alr;Za_m7}6G=H0SCpz$N@~VMS#s4;2LEzfS|7nxrKM@u4 zSK$9aOOm{Xj(>&eAEXDT{%mRW7hwO%a2lsCaQxLV{-85EmEw`V;^9xSWP0A(_7|`G zlNnRb0;KK z^^-FB7J!CvKydCqAuFfqBOt6ObiC*`nC0%TK=GO5j}|8>ODX z%rIKub%yu;GrQkYX8IRXc0KCNq?I~Uk}O8Mo~bfx%H?1CFH}HBQ+rL2cC_cV_jTs$ zI!c4jY<~0{hIL99@@!vql2H^m)Qm*X%YCoq!>5x0$HSIB1{^zdt{HG2xIue_eA?K-i&6{uliv=dWJ*SdmkkAkkr!FT zqD6m244E^BaLCVy;cemr!gzu(-vLe!py(4^o|Jyz06{eL6wRJdyrX3?eVMTd$19`K)p>OZ;Qe=_j@*A@@e#Q;K^+gNB8K{ggA9Vh;q z_Wo6(Y$Hdza5np|Rfm6>EFa7NdSD!uEl}}j;}^S0ALia3&nN}E3-NHeCBW65UnCxL zD0fim&6rWfu^Y5TOoM?x%M=rWhhaiY95{7b@J?&$$5gmRFFM;=NMq%2R`sKNdxKdI zebm>5R(3gc2+wJ{S%6-eddLYw^Jjj}|I0HG-|9ms&*vp28rYbO@N6iS{nK)gvI;S3 zo8b?;ax(i@hW{^>HN?l^K$EnA4tsX+t0v*Xa>$fqDPJenh6#?}q3qD2(6ow8XU>f; zwDq7v0uPR|T8S+V#A@{2KlE-Obd`D_;LrdJx3$~JmY>f!a@Z;YZaZfhatMD@N(ld9 zslyQYpJWsG-;iwnkI{NsC5)1@#)@$6R2)({h+VTOEn}-}-i(Q$KFphqdnkEqM1|cm z^E$r6XrgWqAc`VfrHy*^QRdK;u`(6j^Mvezq6;QM4;{avR8kN~K1wP6|EiDuE|RU$ zm#)X1?O&-VEo%{H=DWt`A_1&hwwNA#y&tOXK-Mp1)?sEElpxNx2diY4L-IlF`j_Y0 zV7^|~%DnxU#!&!WzMF!3h2+L&;(j8ffp=+3${#mHfW5yO89>u0oz;D;PtQ_ApJclY za-VS;iQAwO83)a_HH?G>qkXX}_AKEjqM=M5uPvsLS#7b>YqZiTfV#D}vRT?Nu0+PS z@=Wu0NfVHN(8Tlo8v{^{)BV}GH?U{lTU4Lfp>DngbRaFIO;)CX$?lvOan(S5UiEx_ zU{uu&m`1x6Yb@o7M<35dM)6ibAflP0*hQQV(8gMDlPS5iDzf~lZKU4Y439t-g9+D5v9KbPvu*%qddyiSP1~>P@YM~ zY!sKJcUXdrEwQ#84dWIJVaV^@9L3_4JvSDeQ`gLT;|Y+iYZFtvJ=cCq0tBeA)5j6r zp#^keDBqG%>UXVI>AN&WW+@u)r8E?nAil6sPmX`UB@DvE!Z2H(|9ZrYWbrFgVR zPPb?H>v=+dUZ&AXbKgIqWx!y@peFI0A&@H$IN#7RMt95Oa zrr#QjcZKeoyPAnlGm?Chy2U!YLF`bAM#f9$*GlP3#oVem`^4G?G3=GE3SYf_bqhS_ zCEKsWNqL;_<5I@?+NX?~x~o(7IB(4A^0{XO{dfbkRnvf?W2E%ew9Abvt8cvDnkox2 zeB=A7JhD=65i=+fk!E(0ts~5H4E| z@AS=mHs_o zvLo3CbVk`gPzROD1F;VnF@>NH@H;dl*G4l?lWjN$V1TFI{68QYP@D}9x5GI!flmt4!zExi`$gtf9LD&N`c>(XZL)WF#5h(Q}7 z0QmL~P`lF3h*zEQ6up_G1l2HNW4pXXXdWs&1<~W$Vx5~Lr!MxMLQ{8K((3mvY5neq zIyv?f8?uS$E+qgFTG72#y$ksOHYP-G*l2{lwN+zDg@bg~{0Hw2@765X4bLwT8BJ+$ zegXLKW%7ffi4+n&QrDTkSJ`K5rk4TSK?x>e-AcOgh%v6fL0J{VaE`CSKQb>h=^ zYP+;YGCwe>{_eXcxRhfEW>LxNJ+#8vqn*;5q3$mXw1A{@BVWiI=+ZBmC(mM%0b#16 zBUf6n5{Z0P(}UHLq}HnOf{32$V=k@=z#I&`VVT#=NMqP|@)mTrA;J}|V0uFz8Xajc zJH3e7aK2Vx;D8+u&tu|v7Z0Ba6jGqOQn#LU@IWvkL7y#a0=^_$}2M(a(r)*JhmA&H)&sz zUm(9nFA7*21%Nx)Mzz|@Hap`bkeRRS2uXalzp{dV4E*X2kn2G5Y2MxlTk0HUm9%MH zsB5?J<7qhbXNG|Q$XzB;zZo(Ex!>Jnx?WiSuR1qZ2KMVyO_~!XMM%7nr2SK{BqFo?^R{f4tel{0*rrs5Ea>DV`g2F36vR)YWps zOt@o|N8}W$ExCATN+XT=>KH{rb2ZPm+8yhiw5Fc0_fhKW)^5C>)R4;S>17s?Ls+nH zopi@E*B55AumIUNi=SN<@$2=5U>Z^|YoYIGe3LAN!was^`(hf3>`NFzsQ~Lr`@0!B zewx_0{MC{ovTqge^B*> zom^*!OZYHdbg{?E#2M|l0ITmz{>n@HbWo zLDFJK^Yuh zQl1`?e!4|(fjiXW3Xr=z_nH)CZ|qXu%c&0Gfz`60%Dh3@)J>)F-HZ+UmYl_FK(}j- zLto|O!|~6(b79{EA+O;zM==zkwZr&JY4OPxIRa`)d~7mmVGG{Zn&acF($#`s`UUiE zsMn1EdRxanmKi(JcwU+z_NC$Y8J7X!St<*Bvyi`myR8>S!O2`bCJjCfj;rcSs$iFn z3QWx*)an)L3F!~^=71)>b2W*HK^@xjYA--Y46D|_>L~_|TW^GaDuA2_-y~L7Ok_Ps z&houl!(9n46a(GhrzLf%|3Cr7ZQI$L9MDa>Uh~O>uSZPNho|gFU;NHbZq;%6$f@J1 z9>_1OUSj$YB)4$#ix$?PzX*_TAoRk11h8t?TkK*`K9b|L6u7%V$fLpukt%XWqZjyr za~kiyzMr!!&b>r9f!c`!oq;r2jWd)ocNgj~PF!I@dft9QdKN5`q<_&{UFE(QE>eFF zsf65`BkVLOIAzAxHD7b_9djKep`Oho){2utLK-eBI8DBh2IZOI*-%op;MbYo>`3>F z1}!s(sy9AwIC)=7%vr2$t{Bg=7Lsobe)FYdSbflSwZr_U6>g$ZDf`^<-BKD5r0JQ< z$PYHT10rK0v9Mh{tW#TYH8wvn|H(XRfgV4<@}?fGh4slMywTn`qgpt{CP4E#hj6^Q z>pC`B##cYHd874=Yba$XK%sXBV(9%@V#7W`Yif){>^#U-Bv6hdi40&9)CyA;cFgH^ zc<94M1-q_XIQ)b^z&)VhsumXY4qNEnF(d5n`}C{lnkT7~Nx^Os_ERp4_}*f`s*Zvk z&s;AN=2fH=v4HhC$vG=Ta^?nSnoY;oaMlbi=9$(m$bEM~Ch|OKc#-?E-X{X|aP+v_IiKVwDFRMjYLau@zb>CvqG+CyLul5N9mw$H=I)YNet`>Fj!1Wk z{4#DcIG^T`^r?R{-45`2zMO_f)kRZUl2p^KwNh69)ubS{RDHqZH<3`#HSsznXoQ}a zrXZ>zDJo-_nERk6Lomz(>3Q!5d+~X|xEKn_Tw~r;>D{Mgg51;HzN`Whl&L5Oi>^Hx zYR*+Az1=t5hEZ%FB^;;+bXhX2xe+;XvlG*IaiJTD|Wpq1Uec z=INRbsK%P#%>KU=N&c_VBB%69s4PzgU)cQ@v#ytbDlbYO{aX)j`hR#Llz%?}bu#vx zP8L9;*O?Q=6|0coL3yM$hAy@ja1c`egz`nN4Ws?e>Q4EaoZ}<0z*5lEcDug`L-XpG zUnCP{>C{a=KW(dz%KYv}S(@N3G@dTV$h36h1kWWs0~EqSB;`j+pBA79^bymQ7b0BS zr2);)u+p6jVa<8mRjq(9p*4jeoOGYJRK+H*%hy{9EWa8$*~7~(*gHrCD#k8YSWtgd zoicVXo@Ry(h=ql6GCF%V7fXXW&^BXGzmZV6riHZXRj}LHK2?xE_#$xQtVnjJy@zZV zn|{9Mb*6CW6n32d!BD%@1o%_w@o zbBrYwdfoacH5t``0dO&i!HQdg>%1i5mfF$U3Suzvs1RT7PP9mKDYHEpYTcn})8=LZ z@2CNAF&a7aXNtp_-d$yAj}NjEFq-R{zL*g~SsdPk+HYY>X5by#&VX3!xffTX@5QIW z(cd0Z9&mmaoD10JqjY#2bVgwJ*=E`7RUrOKHf^Y|3lk?(g$@)j$dq*%lxIG~O9Ujs z4Nf5wYmN@prsNF-P!=*N)p|>l*bnbjVc$}|OnZG1OY$oNA+rco_6-2~X+Mf}Xl;bN z=?ul1o_e)#b)m0*cOhi(M7`AD7LdJE7P)%MxI;T#EdhSD_Uf$zr0k2?tbcvhz`x$# z@n1i$;P5`kky<1w&W$aF;ECnNK(25X22#ZGHXeevAXBTI4_@3NPgeoIlfdG$q;52> z&z!WM+($gRuFSnj5+_tPDl0B-F+hTYxx!n68_8^Ul?<;1O;R1~ISVZBaPxQXjV@!w zkDDt)N=B3Y6jF^@cbRNm_MV)NT-mVPwt+NEC9Cc1+crYB*QJ8D>n4|b)=A0xs)=OZ z5#^ctTO{Q>o3>~(UGv;@=ll?GZLs!zQqR%1^g6c=R zoYM8GNr>OxuIwIW8KRIKn8x2DzI0thbQkKN^U_i&5UksKqJePqD=?;JD0fTVV5i^!$lqpJ9%(3OF4XMVOcA)3A%rtBRh>R zKZYT9i3b-jvLmh?$!RmU6-<82v?mPSv?IfXMwB1MiB#XsG#ydiB59dNpeC>RKsGa* zXqA?g=_Z>1!4-qK=?jR4bR`L(T~BaeEQme4m8D-%X_4nb^Du-2YB;R{bwA398}P~9 zGYj2Hfh(I1ZkLjWhPIVW->*3}lB*%br#^JVONnX?h^F$%zoh}?9w*gvxy^(KHv&NN z`pTl443o{Y2a7DyBKcB;*JHWW8A))Y%3a6s2JE@Cg6ale>$#JD5un|gxVaaAO@=s) z7&0bPJ_x(aV&3%nAq;L*8Qjt@1OseHhOVt^5d|$RI<|NH8p(R}qru7iKWxtsmiG@( zeKiu4Z|xKE&AlfW)8sFavOhVBn~Gys=h-s7!KkD0< z{-D1?!9tWWX+K9+*`Wk7Vy7vaAvLuuPKxlId4l74-qlFbQa<>;u|ua8Irp^lfTY(( z)esy!q8wN;bUp2Us{|@yKW@2klOdCbr%`dS(Xf#6HU~|2&t?+~m^3oxTHGtpCjf zE8ZP0^q|e0aW33$k}Pt!(_Z6au@h7!EUUzi1eM>h&v#FCQ+TdP&)a3Ot)*aUdcw;@P2wJnm8hK=7UgN_ zFz8qLQGO=dW%u>_vn8 z1B*p5)^l5`J#FIs%0@yFDm!mF-^sO+kWY?TU)CmuzP!Ld44(qZwJ#tOd!>T$e$psk z{eEtzTaZ&dAzo^RrK{Vu_u+&DNErKph?ID>_^OW&l6he;6QQqYwO=YqTxovJ)OWC{ z%p-c-;&pIPxlWsT$tg0o_QH^;uSw;IZdfQeFKIO{Y-yoSOByRH!ems#O|bSNBYL)U zJoh2&k29A%KP%wpgun;l!lVWu?#_lL`;9r!IaHUJ;knN|}fGZ0*P^2vKFp zTyo@A23)+t*a6AxaJyT3tZ9gvt2Y}Ktbi3tezPAtD8cM3dw#CedY`qt2ogt_H-pAs39WOv9W+$Lmxf}@y^AZAGj}tGqlwIj9wH6RDg(~6MA62Y0 zdABO>0tVfw)ohiw8Xd=Y!DWh3HxSNnqh(p0$!X4H#o9#QmJrYll#i zWI8V7vCDRB{qbZASCE;TJm{rJLeI1b(20)b0tOC=y6PD1`Q#14a}eALIBEHG@kFjK zs){`~yG>DJUCV>EE6oI#O>1xhr;f&8O02%ebvWj8Zz_JTH!F%2!Ns8wU8oPyX7yPK~VR%?Xc zD2I-&gM_7oDu5QNzIK4+)`XGNwn>ZPj`(!I=ng6K`piY3^7JBM1)A^AJ+GvXx26N-EpSM~z0RGoht4Ox%N$C3vv?I)&6%oa0Oj1taFKZ8 zU>$4lqfH7Uy64phDF7kK9+H;*jQ-OFU2+-5X`R}1?h28eEZ2cn!(gGxcr^^%bv6Fu zorCez80PgW6>LzQwRNy8==!6&FV>vaLTSM{OBMV;_*9XUvZVo~3dWi6JYBmj3sak* zKlP;c1PH0whw`4oShZe~@7B)v;akhhaTh$Nyz+4Ct)GvYsrO&B3VG7m8Z?hX^QXW`W2cR$b73NLpis`4g zBknM_QZ}PTN7&?6yS?qvCNzfvMpKjcZnSHgoP0mS_`D)hkJ3=FW)Fx7j(s&o{8G`g|~J|lqzNOc)!ja z)xqLl895Z7rt=FO&4v_0Ji#5#3M5_O$=3ij^8yR|QtPO7#qsek;Eo|c;F?64A2=$Q zv_^8<#+dLh1mvS!GMj`c2_WJMFqdapZ84#k5?(+;BTiiA(<=Kvc8TssM{Z~FAZi`c zUtYHEyAhiRZ{+s|46ZIozezIu?lImp6gU9zG6*1?n4a85H2^on#v>?)w*~68%#inq zlgQ-0WZN1*(y7+zFSfa5PrhQ?K1igYV_^CclUoKhDatI+JHEsH?p^?+&?}d-Mfs0=X1@ZG@{Q~UYXFKfOUXQbrqlG|@nIUk0Tq&1@5eqz6xssLV3)amXr+kE1IxQPn0_|_ei0x z$bG54YSJy$D!TpQzK2@DNg{eXXWV}%EVNguS)^<;H(0UOGM3+R~3S3?u<{+ z?7crBKVio>B#8i0>Y9P>`yJi#J!rni5sC!o{ibbh7O_e&)77kvuD5`C&M2*Y=lypt zWN1fnQ)itd`zYt+E{C|v$GNTB-b6e@UYVetqTdLT<$;A)u6Awz_pZkNjYs;bKts>c j?`c__YYHLnL(r$88v7s Date: Thu, 6 Aug 2026 10:56:26 -0700 Subject: [PATCH 2/7] Update README.md Updates to instance section in Details. --- software/nginx/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/software/nginx/README.md b/software/nginx/README.md index 51ea885..6c301ed 100644 --- a/software/nginx/README.md +++ b/software/nginx/README.md @@ -250,7 +250,9 @@ Intel® QAT is only exposed on bare-metal cloud instances, so this comparison is ## Details -NGINX on GNR 128c (Intel Xeon 6980P), bare metal: 1-node, 2x Intel(R) Xeon(R) 6980P, 128 cores, 500W TDP, HT On, Turbo On, NUMA 6, Total Memory 1536GB (24x64GB DDR5 6400 MT/s [6400 MT/s]), BIOS F23, microcode 0x10003f3, 2x I350 Gigabit Network Connection, 4 QAT engines, 1x14.3G SanDisk 3.2Gen1, 8x3.5T Samsung MZQL23T8HCL5-00A07, 1x7T Micron_7450_MTFDK8G1T9TFR, Ubuntu 24.04.3 LTS, 6.8.0-86-generic. Test by Intel as of Nov 18, 2025, async_mode_nginx version 1.0.0, nginx 1.26.2, OpenSSL 3.0.13, QATEngine 2.0.0 +NGINX on GNR (c4-highmem-288-metal), bare metal: Intel(R) Xeon(R) 6985P, 144 cores, 500W TDP, HT On, Turbo On, NUMA 6, Total Memory 2232GB, microcode 0x1000380, 4 QAT engines, Ubuntu 24.04 LTS, 6.14.0-gcp. Test by Intel as of Oct 6, 2025, async_mode_nginx version 1.0.0, nginx 1.26.2, OpenSSL 3.0.13, QATEngine 2.0.0 + +NGINX on GNR (c4d-highmem-384-metal), bare metal: AMD(R) EPYC(R) 9B45, 192 cores, HT On, Turbo On, NUMA 2, Total Memory 3072GB, microcode 0xb002150, Ubuntu 24.04 LTS, 6.14.0-gcp. Test by Intel as of Oct 6, 2025, async_mode_nginx version 1.0.0, nginx 1.26.2, OpenSSL 3.0.13, QATEngine 2.0.0 Results may vary. From 04429360b61124f376f08726fccbac33b61778b2 Mon Sep 17 00:00:00 2001 From: "Sherman, Srikanth" Date: Thu, 6 Aug 2026 14:01:57 -0700 Subject: [PATCH 3/7] Update results chart and reconcile Details references Replace the CPS chart with a version that drops the C3 SPR bar, so every bar plotted has a corresponding configuration disclosure. Update the Results text to reference the Xeon 6985P (c4-highmem-288-metal) rather than the 6980P, matching the Details section, and reword the worker_processes note so it no longer implies a specific core-count ratio. Correct the C4D entry to say Turin rather than GNR and drop QATEngine from its software list, since QAT is not available on that platform. --- software/nginx/README.md | 8 ++++---- software/nginx/image.png | Bin 22892 -> 21317 bytes 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/software/nginx/README.md b/software/nginx/README.md index 6c301ed..8d85404 100644 --- a/software/nginx/README.md +++ b/software/nginx/README.md @@ -236,7 +236,7 @@ The client count, duration, port, and cipher are set in the USER INPUT block at ### Core Allocation and `worker_processes` -Both configuration files set `worker_processes 48`, which is deliberately a fraction of the cores available on the test system rather than all of them. +Both configuration files set `worker_processes 48`, which is deliberately fewer than the cores available on the test system rather than all of them. This reflects the scenario the guide is intended to demonstrate. In a real deployment, a web tier rarely has an entire high-core-count server to itself — it shares the machine with application, caching, or database workloads. The question that matters is therefore not "what peak CPS can this server reach with every core dedicated to NGINX," but "how much TLS throughput can be delivered from a modest slice of the machine, leaving the rest for other work." @@ -244,15 +244,15 @@ Offloading handshake cryptography to the QAT devices is what makes that slice go ## Results -![NGINX TLS handshake CPS, C3 SPR vs C4D Turin vs C4 GNR with and without QAT](image.png) +![NGINX TLS handshake CPS, C4D Turin vs C4 GNR with and without QAT](image.png) -Intel® QAT is only exposed on bare-metal cloud instances, so this comparison is run there rather than on virtualized shapes. The two C4 GNR bars are the same bare-metal Intel Xeon 6980P system described under [Details](#details), both running `worker_processes 48`, with the QAT modules and QAT engine as the only variable between them — the "without QAT" bar corresponds to `nginx_without_qat.conf` and the "with QAT" bar to `nginx_with_qat.conf`. The C4D Turin instance is likewise bare metal, so the cross-platform comparison holds the provisioning model constant as well. +Intel® QAT is only exposed on bare-metal cloud instances, so this comparison is run there rather than on virtualized shapes. The two C4 GNR bars are the same bare-metal Intel Xeon 6985P system (`c4-highmem-288-metal`) described under [Details](#details), both running `worker_processes 48`, with the QAT modules and QAT engine as the only variable between them — the "without QAT" bar corresponds to `nginx_without_qat.conf` and the "with QAT" bar to `nginx_with_qat.conf`. The C4D Turin instance is likewise bare metal, so the cross-platform comparison holds the provisioning model constant as well. ## Details NGINX on GNR (c4-highmem-288-metal), bare metal: Intel(R) Xeon(R) 6985P, 144 cores, 500W TDP, HT On, Turbo On, NUMA 6, Total Memory 2232GB, microcode 0x1000380, 4 QAT engines, Ubuntu 24.04 LTS, 6.14.0-gcp. Test by Intel as of Oct 6, 2025, async_mode_nginx version 1.0.0, nginx 1.26.2, OpenSSL 3.0.13, QATEngine 2.0.0 -NGINX on GNR (c4d-highmem-384-metal), bare metal: AMD(R) EPYC(R) 9B45, 192 cores, HT On, Turbo On, NUMA 2, Total Memory 3072GB, microcode 0xb002150, Ubuntu 24.04 LTS, 6.14.0-gcp. Test by Intel as of Oct 6, 2025, async_mode_nginx version 1.0.0, nginx 1.26.2, OpenSSL 3.0.13, QATEngine 2.0.0 +NGINX on Turin (c4d-highmem-384-metal), bare metal: AMD(R) EPYC(R) 9B45, 192 cores, HT On, Turbo On, NUMA 2, Total Memory 3072GB, microcode 0xb002150, Ubuntu 24.04 LTS, 6.14.0-gcp. Test by Intel as of Oct 6, 2025, async_mode_nginx version 1.0.0, nginx 1.26.2, OpenSSL 3.0.13 Results may vary. diff --git a/software/nginx/image.png b/software/nginx/image.png index 87d92b55f11710e3ba93fddd94a4a8b4fb65e572..7abbd2b196cb8d8d8fe6c675b7fe1bf083810360 100644 GIT binary patch literal 21317 zcmeIa2UJsAw=b>-D+s7G0R=%K0zxcEFNy*}1e7WzL8S>Oy@nDCMY@C%5otm|R6yw^ zAp+8SXi`H-fKU<$fdHYrsOOw-e8=y*@4f&1-|_Ca<2VMxy;t^JbFR65^S9<&d+il= zTUV2nnU8tTo;|EuH*e_g*~0{4{2e*O#3(6gYj%|5$pI@k8> z`5eQtb$>tO{?YAQhU(vb{2Lq`+}+*1fB(LzscAX!oq^Rz5SUpXGTXy!@|N!N=mxA zx?a6{1%*PTrKNLnax^tHuUEAm{ce9E~nYGZA?JE5RMLIsns-R%L!Ju~`7(}gZj&=IEpq6@3(Y)rknU}DvdK;p=n z>?e_#&6#Kj-{kvI}w6NOlxR0#&ooT$+AF1G*z03w0X*Dq0=HTapb|Mq@rSK zeZFM~(W+`}O;)N0U=|?I19D^Dyer>WSC!!JGV5!(ex-1|`J# zCME|>405?tThh65uC-1~vC^?=eyUsdIxrVednfpC^R0?3mB31J$@>*zlfnI41EMZHUVRv@FT^NT*&@BXYctkscEqI zjfLxz31nk({L97^`z5B9pm@Np2T^KZWEJhG5|4-;?)#04-3y6;M7?=tgz33VW^?xI zYU)5!eWuLzk!~n&@b{?~;sWGyVwR`iARNC8E5r=-2z zM9bol6Nx%9v-El|cP8-J`RZLxyVQ{|)77$=Rxl11deodq{w6sZ7`WJuPOVeF^R3l* zl88l%^=C1ux^!e9TEH<6RrdCOIA-G+7&Md}1mL1$7FT`W2L?1P;5_?zc|PxBdw2ca?EO)WIXaQwroK#s?*2kkaa>?Ej7lk0Kp?Gn~oJ>B{)oz)10pkYrR75k;xYk9{ zhg3Jhb$pHhga|Xa6#6U~D!Z*sM|vlp=`;9tZmJwnJpX=SFki1V$EZ8GWOe^c!$FvS z<_5@ltRc?nH)3h)reH*Ab^#Cm8Oncwo4sR1-(Rpq#NIB%vTCg+%Y)Rv6Qr%0y4HG5 z6h$o`OQ^q)l317MOhi5hv!jds%c~YEhYxzsm%Icuz8%gV0nnDSF{U0dt^6iW^GigO zokaFMs`#d?WvO5O{N;rhcJx>b%uRG@?JOA1?{W&bVVj~DApv{3Ho!068H^ZNG&*-1 zTPWT-)3Auq(MjPTnB;HnRGxeiJ?fts$pL|*$aJG>Pb0H_KB&?9`n8BvQBxt&FMU;- zD7=Fj?zU?@Qr4&W)L>rn=(Ha%|(RD!-e{&OS-A=UJP$7bdm2WAI7ehwG|EXwv1=CKPxT*;Nyc zOIwGVK((9;9z2EjdpJxS&d(0KDY|rXL&v0034=J8a@0OEG7bVa$0cv|V`L;pjY)S4 z%`FF-hP476Vb7tVMKW6dhviV6#OquNG6af&fCS*{9*AA6cWu0IS zHrwa>KCwMK)QP@x2bZ}F`M9FrZ8BM>4kRXSAaSd=PC~ULqO3_Jj=^aZxSxK?w3qAj zl51U!n1r`7Ro`EX@CxzPY@xC0@S*|ZmjA0CQOe``x&ZUU%`>nI`2@gXgPZ8Q%u0hR zusP3N=?*j`{IZKgwSUWN<;E6g68u|CD8E1%))$x%AZldqx=xyx^X?g4l|c7V%C00p z#08X8E4*}#fq9cDn)q@=%kYr_Ii*E_ueosh*AG2Y6Lphy2F44}S)>uHNEd7{{_-v% zM-gD39=zaq*t1xu zq@*7x=#KcBKd)!QU%-fJ6rzP<9Ludo1YDfZF5Jw(1r6sZ07q>*XKH)OX8|VnE~Lern^Q6eFYd@=a)o_C)J9chVpKrR$*T{m~m?Fz72 zLT=~0l)jN)Er=qAOL?a$f1S{yE}tNaRSI7TC738&@7Y{81I@0bxZdOjthLC>{!px$tnrc3O@_4ESw`^(0$wu?%G1Wb~eOpriSiODQDyo>3^})br zK<95%LK`E3!Iqbnvr?w@Zl_TL$`6T|u9a6$FTIfZM5I?WlGesN`5!q`3!@kV8I-s- zP3*B62!Nhze~?|TywiwBxD|n&wpNE)W@XRYOqSv zw?taW%IB!ou7DE}Nl#Gf7?fIl+CJ(e#w*n6Wz%>aSP`mP_!JbOc<%6Q=fi5lu*R?| z1vXT#{D)@8&zdNa?Q<0WW=wN6g?y&fhM0aJj^zcFCJNk0Y%C-bQeu`4IG@R%0m!Gj zaH}p(wx{=xAB12v)trLk!?w7juL*>$cSz$VPNHUZtXWE=UZ|d5((5;3mDT2iF#9z$ z&bd6lhXgE|tfe=(frV9_+#+f#uzs%IxysW>l1!?K@5Jz;5VXVh zl(f{=m+>_*(^O5bfb}bBmFixVVPQSAWv4A?$<^w%6x-Zbl5wRj%lgzN1Qt)kjYMd^ z9I7RmB(syso{4FWkPp{6o({CHbdocVuTq$-OZB*mRT@-gFn+gG?dK40ZbCxUC}p|d z$Y66Wm4zRcJXwZ9BQCr6%|!R7DZ=9N$y1A@x?usGha`X-tA4QV7%SACy9;#Y3FP&D zD)vx1!geoO`nOr6=}{<8lcE~%*rg-X<3e>;|Bo(G@ICdaCxxIUg4RWZb%+3+zojJ>D6M^jQCr8)&D|TxhZMt4y zlbx#;nD9lJHxIpRSmGKSUZAzO2SF6-&u+OX1v1}JH9s2hcK3<53IyrjQ_d-NxGF@j zLy%`J?Tl14WgHXmvb33nAiUEwy8TJ;!0IirHT~3=hTH2JUCe?SPn=T#s+Bhn)9ws}Kxe(#>N5T<^b_oe<*S$E{3gIoe^lQjhFD4!B#AYAh;OA; zVD$sTX5YD#z0^z8Zx(bjMxzhI0*e5=d9B5ieCy&mq-h!)g!jEK%1YN`!f?Hj>L+1-Zp+S zDtoPUS-dD_QV>bS9=^XVZnS>x_O~?@qV}8-;i~2Yf8z|=1Bd$<_h|j)$N?$liG5CB zPp=Cjv)LHM6VC$phxU>_$>vn2%ka+v*qxT~pnE&@wnB-Dl1AtJE#qyomatkA>yH~8 zfLg&+Es&$rlcnH}70A$e!MI0CZyp_e|Fdfkh89j&;l8%L zb>M_3ZHk$eVtmrcLy~5Ha3FRB6e#`KbfpF*Vrd_LN8KXV5B}LG%hdfVDC6E_S=;pr zKaW1`ix8XsOG?@kNP}9U;}aeqE&t~U>gx^5A5iNhJUGAx=q^pR9Wwn5c;?pUJJ?z>Y;T4m*Dev zCCHk!Jj-C!v37Jpv4tvRnb>-cUzH;|PtMzdB5y^@ptzNH=k;gG;nhEeZ|)GA>>6hp z8!7tok9_&`Cc8l0oLw(g2u^g))Ev$!Bkf#;dF01UMX7}s5bb?HdA1?YC2b>L0*1xB zo-9=GmNnj*z-jh=u#vRflYJ5b*450PFgPBmIsej|PmWALqFmF)vqu&$f(`kwDS0H{_^FYWgDY0r z2D9ts3qc4h(_8=YJF2$2^1?BK1@(O(cUWw@awb>1+WDjWi zrSp?*&PghXh}tJn`}0?y#t^KMt$A7rc^(%S^$8WH6}ou_VU=@&rXn2sAe1wXEv=af z7`GPKdrFg4Cva#4B*6H&dMMXxP0?wQKz9&4YMzo0F z4ezVS@x{&1#>R!$RV)Dw*kAzJp+(}I`FU>!rZpJ@GZGQa0>np;ap4_+!Hv~{BTGIj z+2_5SvOub4OKt$G#71xT1d*!vgtB_}bMdo)ypiw;jg?02j^}a;Xw?P9cQ5R|!@7TF z-r+ZZ^t-~eqc@*Mf?MvVE%m%8zQf@4Fb!+rM+*S9AZ10IzTzA}4ad@8kl zsF_U`YZL$SgI2fe>0`C*mq;02QKGkp)9PI^y?u>7a_RT;WZ0d_>SyOsGFJ&fjEuvd zRmaRUuK8PNcUQ1uxQA9{P0zqmm?={!D%Z?51&UD9491nlap=hsodkjsLUjzA)J|=2 zU!dtjRW&VZ;u=f$%HLw>JQ-u4DXSB(Nu+@%k-;C8PO=&5reg-MCSO-3G>Xoqvh}4X zj^#0n5|;Nul3()*)`1F6?*tN$b1zkkZ`|wx<;eLgR{S>QU1HtOxLFUfLYRzOW(x@( zXAF7>{^vNM&8{wx=tj_~y0f%`@_ON-rx0Q56SDxdkpB-C$oqpW%^Jc!9U@rB$3(BmGrXvpEWmxxpUWS;heR%MYZ~I)}*1NgdfV&@fFu~+b0)%<0 zuHVWptoRv(?c=mTZjm&gR*QyI}njpY0^AX>l5)cpbZm@I?l+?al`O^F_#JZJ8m?mgmg~vtjVUl zQGyr0ik9VD`kFZJZ%})=#Djv<1A4p(5VyLI@i$#@l~Pr;AbbD2go}|ppWyEwX9ffh zhZa6iSVUH1a9$B@Ag3A!Cv3qT6K!kGzX%-P30e1d9 z0V)>F+@hF$KOGUmgcdo)eh|_UQX1*4PKr^v(LU^RLFFUgZB9YzZk|%M z!lxQu;D+Q|#N?*cDt`qd!Ic}2`fbriZV4_(YIctG(+a)w%&w&C@{-6EY`@7xbKh7)54Z5UcO9c26ZiamI$Wtjeii`8is=kpH-yROBJWj=aI*$ibBwsj?Z|<>f z`!t$&UbDR~pLFj<$^2@=@_mlM%O{sD0e>p=Bb6t;M#5f7H-Y~sg%nu}sshh| zKU3HZxHoTGPYD)gRmd4RU2URYW(GF*c&}n*l%C^ei#AGN>uJ-Ih825-$eraVPV~fF zPH>s=@3pH=QLu#rg4>VKc4}7=zCU)Y1{Q>3pQ6CG23M)alY|*pRKQ2N=cjObpZPze z;8m7pRL3>tm(3TYfE6;sg*b&cxw?I=O%`v&Wl!U`c@YTe;RWMt?@i-_PM9Lf+AD9{lb(z zt%W1UKtZ?mLBn1$V0?)8z<=hVX<-8saV=PU7I4Y0OHAh7?fvK*w(JV^2e{s+BbpYj z<$Oz5<6<;gd6tvk@ZB?raM8h(2hW*}xWB>wf-k zQ(1VdkNu!?Ej}App&DS8*j}IO#-yhE?1zOK6UfK55$XjHv(-zy+}S^2V*kSd_2*E< z!=Bw-RqYOfwktZ)N|L{|Itgro40RCg4BueCadHT4j8uJgC=2xiV1Q~wxjgbA{U7HZu z43b!U--;^kPxxoWQoz+zR=Plw%g+j3eYIl68cz^2)x?dsG+-_#N>-W0HaR(fn46&$Ecpx0Vn4GK~yneu^{wAzmp( z!W+3poO{VG&=fKBjekV)L z9{`1a_O%}^AObFl{NN62YY1i3b4=V3ocK>QTdgp(HVZO%bK<8es}=o%)hOAp%ePei zNl_BQtL9`CcorF#>!zE76;|u`vt=nLc@Z)jQQ`(kQut2}jfprU&+%9kp z-t7C?J++|WhY?0NPnM);4#K1X|4}!-KX`c2E5*H@f##^9D2_^%_LOgblIAfq>)yvn=?XcK)4vTR|QC$tpVM>`7l4 zE%>vLlGW_lQGYl^eC8fQ_D|S?3!2mTTuDs~k)NY0Ba^YaKYD%a&`7H@Zq!HbZW+bW*UDBAOHW)VUca1`D!vf_|{_^2mZvyisJH- zCn$-#+&Z_@#ZHVc61o!zMB-#-l$>=pu*!)Gb$kXp>BdBNs7e;>fzye~H zS6pk_n5*;IKTc9g_LcIhL78{oyYr7;t8YRT_h>&HeT?iUR;HZLI6w8hn&%tOlO8{3 zW_0YIogdi>iU=HdOix&e^Z7uF*tRgUaJm4yQ1tkA2E&|v`(Hn+VO)m0`sSKb(Z+IP z;>oq3st_PV1*7Q(m0l$ zgGYk@wW2$|Vjkz3bU5e&%)c1%97fH%wP#4qIyBqTwT`0nzTzX~*v0F%EI&LZuf{Kb8YguQ3a<>3Qd z47dLYh;&5uUfWVuaYpne|076snEvYEXOPzi{@RHBXld5J`jHuG&-aVxs95R04gA}K zKgWardJKkt&pK%VnAu7v369}UavjKe!U6Y#e}(>uB!tXdX8t_)cHz8njfAVzzb3nK zRPpRUphF4vy&aRgy=z6y!mpu8KBOdjD?wvA>3*wIKvv9ps70LLB`UKVkEOMy1Sbaw4YwE!#Ga zGSZ&~zpHS(VGzdZhbN2g z#hW>J8{rCG{m!oy(y=-L`+iXv>q~q|2+f6d{FUmzXW{u6Qr)jZ$|$Hc27AZu7jlan z1$}+`H{<8@{vOoSxH1{DYVyF2GtRjP>GXr#6#qJc*UzGg-5_REP|_MC?B7NF zG}05p*qCH&w)$uUD~I4l@V5#4m3SUJf(%LWOd|7o6~JX2zq7}$45u-Tw87;fsH*mL z#NVqD{G}=j$k5t6BN6=Hz#Jo;A?xU5$7%Sd#HQ^-i> z@CC%#!rYi&)u;gXwz3=;i zFw zp^J(2BA}#^u;1s3JHHSBW3ALdRIn3j_3vw6|Dy+zdJlB*0zEL8 z#g65dgvOLX0FTSI{-*-1g7l!r*)=R8Hk-T_<&cXh{W<|#=_(VX|zWr`Tj`gWm4}&U+4Sp zA#PG`(c-+ciAY7yTVv)71(imyxBSk3K2ZpnGY){^o$R+}_z{uJXUVL$7*o&_e+Z-_ z@>|Od8u94;P-11VF4&93cfl6>){`>UR34`Z1S2?d;{;!>(>xGpKacE3HWgl``6G(U zLol(j@a8N0#Tp? zsIzpWqAwNeDgRnwL(=$Oi8fMcSF72fgySFp_&8?mfjAXgb(NtrGq+WH`z))m8nt|Zgzp9o5bn-VWHY>y z4Xxy!`xq+*D5Wr78Q1FXI2nU>={J|t3J`cUki_LYng@Lx&<|q$-ulnIHmS#~zUt>Y z-wcL?_2rvWTR)P8dQB^>%kGAClrF(Pd@3FLU>4I2V*TDy#^VsUz9m%bBck$QUk+5v zQ+YvV)T@k(Bi#2AFl?B6TX^;%%S(vEL9_=r8Y$(gHR{sU5Wz^!lY1K?a8ZK`m8;pq zl8gml!aIaV6&Jdfo8kGBAn;u5us!1Hy0NK`F(+CifYF@gQ@+IfyWYtJW1f?%$LEob zKYhb8dMMhkB|uTiH+SB<8{|gA&kj_nmqGGIx%qLYVo~viFXVQS6%cz-iN|>lmm8* zH4dRt@{XMs!o+BfZkDd!wEskhxn;z9>B|+pvep2C_gzQ%t}A2LHsLoY@Rq9Mg>!_| zR-oT}4E%~`zL!Ng-m$=H;Q>{$e+&8U%T)+m^x@8h{(5sGK4W+(@d~`Zg&z!E;B)H$ z@tw(AO+bmNGInTJGQf-gbY8gz_AF|`g~yUQq#9q^@e|YDcPPdDUyBw}?A2s7KZd=AioqHkia<7sdV{t( zk|TKJxmbLk!QCdqo-LC;+tEQ z&e$j=K91sr0%XuCIiLV`AseYH5h>)fW)xohyoqJQiQEG$k{~ECh1YvdN((X#s3te_SA=2Sjcw zH}KY?z!ClW7{jt#oc~v$Khen(b z_GR_$#B&vloeG9jN;&JL@GAu#OO1Q9<&7zZHhQC4p$K}xcXTHOF*kDGI!%TeJ zjNZi9E*JV;LKnP%_3SsLeIC_iIGVw(zV34mR6mj9&FVw5saVD93{=bJNGXvIv4ij> zw+3G3AjWtZF@}=&L?VgY;cS4O^tVRzBdwn%@Ctj9Y=i>Ldoh%8 zQVPS;?`^9?cJ$q1x@5_rEW4 z=?5`B@}84l1dPCsx<$J~Y5@lKMsfxIFRJ*5q5+ydUr{FOV|``LDcw_XG;YWYV4Krl zWwT?yp>H!QUohxQ)#VAj9Ox0$Sx<$oXUQJdFRypk;b%abtMPKj8g0*cRX!OI0kWWa zQLI+U6~L$pp$DGp-yVr1UFP)(0maQ=r;Z;BK_9Af39`P624#`YFyK=yM%aFb0lh}eAKX#A|g`6NBB#KKA8v{NRM1z*+iD&DkA2Q!4y}7d#-b*y?5}KFU?yorK+x++#^R<&ATlXHNYfI^+N*qy8 z`Qn7&(nR>Gt8II4O^yHvy{@JdeW+UD$m}c}e}3?yO)#Q`^}SCNqVUzww;mf%()k>Z zyi^tY!o1POkwhoj8AWXy{#%ow6+_m1+ip`fJ$=IY9>+Z9@Dq{GOtL!CfOgR+Ac5)pS0;x^NsAd$q}>?QUbDuzoBH}MfmIgQxYE(e>l z-s-$S2oZ~q#nrKOGrnN$ja_7db_JwK*Oq65_qsT4a2PinDv-yS5U%9L4V8WTRN~mc z6ZbJfD7pbPTQY{J4OaDk$ByO<8R_*{r@T_axSTr98Nsbwjqo!}4P)hVjbZ%{{%NQLFUbD)_<3(id+ zm(qm@xR*pgnEPFurSy$u(pH|?P?&+phV{>L02;h=-U^~nXazx@1=i31W~+|Tl9*Wl zcoUhcn+g@cJtKzwtfZntdP(hsX7`|-{G;8nGZwYIgBjjRuH6CXsYyJ+?p}{w06UwG zS5o06qjvi8gDP3fqC8a}6&A*k-&D7)PePX4oVzp_b(A|YFwWRE863!f1tE9EB=R~C zX)&M^K67O#h8CV$*czAf^`=Tf%|=JhYgFC+rdGc6@Q64B&+E~as&lDwIKu6$%-Ox@ zg(KAsplbfBP;Q&r)v;61<~(rY4%+3~OV;Rxa?{2JZLic!4KFj9O8|AzKTt@>bw{+j z*iyfqw}7=5&T}hBdU^6unKa|6<{jRN*Uv934+PnEslA0Pri>RKFrl$!K_UJ0L2eQ zx5kNr-Kt$GyQbsL%T~-GE+&HI=Y20Klki_{NThaAMdbO%0Ijy@4~YZ5 z_HB`heLpFaF-lzeF*kY2p#1-2QHNjHQTUgvtBSRMNs`HEyC-+W}R)7|bfgc7E(=D*k(_pa$BYP_X_(r@oY0YQN%=@@8y$ z9*}e_Y2uwlO02++$->@8CRhW?^Y{Fh(UXjLY>9wz?=L4(i#LA2p z(sfFma#_prhPPcBof7z*IzW{+U3Y22r>o{xmgZ!6o?0-YJ#4;!k_{gsLx!{uThg{G z&0jEKmPq!UWF^*o1&A!19>)Ovq&9G6kp-}PfH3uQu&d`nrym2CI?XQukUsk z>Azm(&Nk26YLC>F^jvhxl)CfwJJYm$Ooy1!lCv`0_vTJe_3rYXdD+THL)x@}i_`}D z?t*|Sy$^|UfCJsTqqY^_1=`vsH<$a9G`b5pd1$SVY$C5pJ%?0Y%0Xu#VT@0j-7fdg zwp;$>6dGtJpZ zM;`wOCF!BJ2)M>HAfbn~_NTyq@j){G!4WmilPf#CJm^#ekiI?O45n@S?rt%eR42@A z+gWgKRN!`NZLIK(tx?r2W;fsOo(CFp^l56yBXYl&@ zL8?{37CqS+Lb)z`WNVnblF(8Hv{YAdK;4v1*_^=#r)>xbfYGznt!d&C;?O?Ezb(rvb+VXO^KPcm{IT@0^a=^p17?S(^SxqX7}= z;Q>C96{hA>;i`pcmTwT1PRmAXe!gR>z)EL~cJtZqwzx#P z&-?FOxgQPLG~tYb(a8)mObiupG&+uMmwWXRP^<4jd>-Shhyy85fX)I9J9#AP{RB#6 z_3mr)g*4c^FPLq$tgK$VWQF^CBS`o`k#dLG(5&%C&8HHDP`xi13I6a{McwCbsu*j) zj=3+H8i29%?@PQ*LK>C^Rt}2|w$Y?#12*jj!tcJgTpch--u-%Z7fJc zMJi+*fUej)LpzlZ-s}W{w6;ZQ{%#^^G)&+_am=o3!h1?tImd3gqWxq`4($E}9N5lLxkos@&rVoqU9Wos~}5gwKweM7QiaJVw>Eg0k4U zva=DO zO@M3)fO0WwyRM0F{mlI`G&pzDj0*w0h`!=B4j^hTF_63F zW=91t5S>ioffd*X7?>+$6SX_CJ+v#~lqkI@^(5X;SrB{NxEJK4!G^`{DDB3tENP;~ z_o>YEx1*>!@OT~@;DT9~$}^~G&ThqUUodGg`p$3bUvKL0MX%N4zO{@kYpi_%O$bQ4 zt5fxFA;);DtXxHj^qqN=(M4-6kh*gxJJGBOg=bEm7hGRNQ7Px`q*t)G8GMv# z;cf)J(6+@njmMOqJTH9t<@u$-%>C_GmMck)kMLX&3XKo;%(lNF4!$OQcsq}-1o93t zlmNmmo3vcnh)G;ht%e>Bh=oNY15EOF0^%l#EopDUX3`#<^rn^VZVL-^iEMA2EsKO% z9Nrn;7|Od@t|((-lv}r(rM97Wi?&F!i}lnm%q*W*cQU47ZqSM%Hq|qwDoGpkP7FzVcS|>(i`!=#Vtiq4@!J_{Xu;Tax z3g7X`Lv4Q4qd!KJhj4CffJ?940?j3c7TG?LTX+-D;YTcZJr6>;aH~@aJ}_I<6;ZJ~ zGu5i_VKW8KD4A}h&nCAQfW(#Ga2OJ12cC@VdZj;dWw*fT?aWkK$YISh4L$Rbpn?-p#p|+I7;Jjw8+Kk>}MDET&BTCK5{}fPio&#s$;3(+&zzur>L9TVQ4VxK~?)e zK_9IhPKfp@T$+6p-5hgQJVKcFwf$?x(t-uYjhjToj4{PSRD9c*Nx7BJJA?h8lq`sJ zPwheuALzr`Tk}XuS50N!YJ1*P4`lIM1gG8UifiGYQ^5Nz^+yvL>S^JxZ3X*Mv;%}^ zZ)|mI4|gx?dBqh$ztN87KM?G5!R&i`B|N|9?-JGrctMZSIYx0yxlr|2gE={ z>*GpC*qCZ-UMOdO*QJZ7_?GziW0qs7*UuE3gWuP%VJtJj@pFLD5oJCW6IV=;VHC{s z(h7X!eDgLPQ;z=vQcXb>N4Wv(he(4aU!UFv`&=0mR!&QpP9pbc?H}W^5#&R~N1Ny& zJ&ubr=A&BO&bb}dWsEPV{hQcQ7@uPbPF#78x^ckIUN-+_+JrNG8^rilOz`8xd$z@} zI4L`CXV(r;a}(*TxKdMO0(b&>8zjKk2QghASGW~w$Uin4Xm>2KY&sA*cD`AcbS=(> z!CBjDqmPpi59(92tFNcp)+gPIdoiM4ncG`6#Xj*6y$=G{?L;Vf`kG%PdCej}buUAk zjUuU7eji6;0lpUH0)w$197O0%+{gQ|`NZ;U0SfDm0@t^Bphf)QI- z#3@~|{lN(TiS_If#vw4?qSH-9N?bZY&-pJ#|*~f1G_;5 zn%eyZh=n`;fh}1#{Wk{DeAJ3uqE-H)N;o6FTHOK@@{ zowzAt-P#6%vrx<$O%U(W)Kwu(dj@aPm%|(TRTO*uu&y_NXug5Z@suh#8@N?8kaz^N z8jCViT1?=7VM&VlxWceIlyN-IlHm<147C+*va#foy`=x~_E$C};z88(gVy7)P>w3s z?fW6Lowr{EX^nYoTnkSTYV#}do_idyYVeVjjPrSvCiBDOqwxfHXXFnN%5lW#K(XQsRJRn%zWaG_TkfLniT zBzf-yg8f>Xau>o#XuA;q!B2L$y>)oWevD;{LT`P(t-5^S&HLdStN5gRmFkUDuiniz z^v5Mor|t$@zG|U!zGlOUaKS^ag8tTv_5nGEsbrDzb;zAGh*oamj$cVA3bWpbCLML3 zS0D3Rs+g5&8(UuE?LzwxeC)vO-{Fl~94qcR!`gU!%&)8B8tl?#17Or%`!zgbHFmiU zAk}L_{-PM-#DwmRuAiY4s)&!UFu8q@!8=MLt^#9y(+gG(X~GV8urfYlM)l7mk*~ygoBQJ*?o9cEo2WTp-kh+#rE%-Xk&-x?J&Ti+ z_l^(Mb?=P)`WYP^C6mb(78W27sJFK_J3BixG_<+7IWsdeFfg#VxY*s@eSUsECME`h z!Pwi|L!r>>>S{vJH3I_!N>7Z8jEjqlc6N4`FJB%XACHWTEH5wr^5siPN{YX~zoevO zPEL-Rnwn~D*V&(w^3>MUfzKGl9i_aXcD)DkIC6xA@#pW+PX0CJBS#Da@87uW|x zr}lw87%rF$xLP4K5+5&)l*phZla(jR1D4_+Ks_pA9OpqI*HoBKda}^bJtyxkllL!& zZujjiLuBusCK~K5XF^W#wm|NH&NWei@1N#L4c{7hlbea2s%}W^=LV z-tawXWn>1y3IUV5WyM?v96_-ntfj1Kto^II*rnFeGHz5YNpfsa_yVlfjnhB|yE5m` z+5E|dQ^xLjzkGTIEI))$(J&r^TT=C|U^Kv0VFwsd{Orzc&_dDGn&8@EzcHsT`M7o@ zK)Tkg4x>;Hs{;l7Q2Vm$y8l^vk||21+I%7gH#pBZxju?1ws3YIAZ~ z_YBn)fQ^N-L9rL|2 zD^)(`0!^O3H|}WXxV%g{wD!4zsHrAjMT$J;}oWOj|K- z{;E@!0HrecOQ!RwN2Ee{P{g5`61g!i+gP!#dGC6zC6~4w zpC%Zw_h5j3=ds!AS~ruRvYyHtCYusf9EhRjV$;ddSbVdr*mB{OCi>-jga%At9ZvzZ$FU_gIKdc#X-wqTe}3{^m=&SSTt_+GS<1) zbFnJ=wW4OmM|TBarCKb+UsJ4$YP0pI*I*p5fIH`N4_hIZ`?y_E4K!?csdfoQsym(U z%;4r!%khHckM4)6SJ3lL^!DC8X!ivB%{e-~>QP7R^c*N#yr|gMc=dJ?4rIi|%(&-mjE8?=^zz~uw!bz(ABA+&cObuAyw280DC)E2u`2%|Nt z_s`a-qJ`YbsvDz_BTnm!(tDt~2T=F9*1=l2Cr}JPYNK%}WZB1RZpvu(alu-{>)}bP zp*zF0#WHvwPhGW7*(_KK&i~SRD5$61vOTCY!CO$Cj z8=4Lo;0EoJ`gRR)8fhAPRSva}3l_?H(`@XpJTyg}IU(OUqmRy6Kp#YCJ!lu+aJa2E zFI4@2SnkoPXL2-OzbA3nPx=vy7SrV6nYBg>2<$V^) zZWZ!~uXmle2VEMjZD1ejarga>wVk`^p905^_iS7>nQYZTgb{9BX!btuQ@uqI-LlHu zto4*9mpyvF4e+?Lbtz>Z$lUB=`!V7qV1E&WMDn2Y`c}5U#d=%hiC>vX+ z4W0JWGD+CnyE0g(1e{vW(-o9YEmHHn>zKPde@a0cvXD_Ax{SXia6A3uGn)?0AOuE- z?2!x}!LR{6<~bNvz4a%u8hNdW_n^~tHNq2v69dOhk<4O`oKQsP^Djt0_MB=RqXrF+ z+f&`065KO{MlUb~!6t0xS{RYf(9KqL7FA{wMT-Ez!7$nDMYbLHF|ytbiLqK153ubp zWbaV6jBTl#$CW6S=_DkxeCBj&8B0K{jZm6d5}g-Bs_BulZ1sqAiL4O>f#`Je&kdqt zl4g=hYuA1y8H4rfvK-MXpLq}d;m^6;93&|eH4DyN5V`No2ejz)9_9()r!Qq@k6~Z( zQaED^dTEfikYl>z>RUbaIAH_b3`l}$$AWXa3)gX`c$`kRb~VQp@EpI7>Id#Kxb|BO z9Ec?NmzeW1LtY6nI1o~27May_VZP#1Iet~uW5lE2ph8G!-&b~uD45=>J?Y~&dMNQj7{oWn`fDgDyHp?q)5fFENN;;BiK3 zQv7Amx4g(y=hZ9p_hAV>`J3ghJ|%in%V%(914M^%`)0xsNdgEc-$I>eq)^h7FVwNL zpnn%@P-2U4mvIg^)d^RdUk-xBqrLV^l-m~g?Pb&94GrlffJi0ag{GwQ8{<_SImWZ# zxteOFWJFsMp!L-I#^qPmx#pGHJTU4TH8($;mVs5cS9NIG3T@eO_o4HY&0mh4Fi3Gv zmIn63OV!&!#YZ^Ln~4IwX*4k}qc8*UfKmKjRo0Rd&4vMslaATw)T5n1Lr9@SS+KdY zL&rEbiZY8gaFRPv_uCx!{@C)a=7;nRdBaPTgT z%KevIeP|7mkk^q_z$oF?c#dpWQN%U(;tv10ThP@Pm^hZk80rp#%e72c)>xcc$X0H@ zShuLD*@J*ImYEG?Sr1i9iB(t8$xxU?99YOZX!KLW7rTR_m;|?O`K+5d3MB|R^LJg2 zZN!=2s)m^HQ`r1sY+IzTTB&Ey#uRSP0?o3@w`|_`uXts{>1WgG zw{zIoFwa+;pL>E8yXge-JVBkE?jSDBvl&LvyoG>=j)AOJ%FZv}uU4Fyopv(_005i4 zMgzT@eRM|@uWyXi0Pd$qSPMaH%L~Rl&W7Ou^<|S@&jT-BxPVs@SCNrk6u%s$YWMvr z5Lhc@A!A0M_>9S;;)Jd;yLeG`y9oKIiEw^kwFW>r*{>gzr!3lFK4|`|HFGK^f1Tq+ z;L&dtoU`UAj}LW?QI*Qfj*YNWFmujP)WC&-*CnapwN>78IUc#DH&Q;HZPW^h+*omA z66^d{&!i;^tkD3#4BU@b0)%L`aHD99qHo3N9aI99aw?57l4A|Aqhh4)sdqtaEY2q> z*nzFLYucvb!ko?d2AcDG!9g6siEoz1L>wN*h2ap=%hN6=9U@yMnP#vz(IYjJFG_yE zU%Qv&6hXN%7L~>(!^=&-d-ZGqFrq%S_Vd14#wYh`HP$|-g8$J z$Kon$dwc@wPK#&o5!nY$nW~2)*To(}#g2gIOy2@bk94AIZ-Q!$B*J}eEt#HWdIzAM z3PD~8W$27Qt0V|i*?#-4FPy{UN+GfpbHUbpHsQ$I)$+bgy*hykfr;=GhGiuno@I0& zaVIl-ufc;=2^06UzfZMaD|?OM5ot?rNmE80uU6yNONj2*GJ;J8PYj^M;pVYJJWCao z$RQTj?`mHqm6er#_^H2Lah`$_W{O9_2J;CEU8~^pX(6+$fb- zYK68~XWST$fiwlr@QUm&IOq4MxmjklZZ(AQ8^~5F?(5t)ulLe%tEBBZ4Yc~|@#pPK z0~2%<(T?uEEj-iEbfBbc(UehZS-ZWkT?AcMuLS$MYHc+5H1{X(c)oF?n_j<;@a1VYR2?w^MFMrr3cGFSpYuQmvPm>c!Mh-jse}-ZS^k)K}kKT#5lBV zWVftBlCUzfpjN;Q;Et%?epKwLM68zWoOhkk@aPOxhtuPnS2EwErEMt#j7Q5*As$c8 z;GR}tI5tm|`-C79KM5&)U>{K`85P_R8xT$-IvK!Tivc~`@T1YTwjJyr;<73#j<#$4 zP)LXOtTyyk6lkuUOFM|#6RdHDLZ?L^G<~(+=78c(Wrl4Yq!>fd??0kHhbz9pnB9Jp zd1p{TPDByhv-xHG_5B36ULj%wtCM3r-s0H?*OOHSZkBBZI9~4Sig!!l3;Z6ub4e-Z zytzzG#ada&{yi9cUAn}lryFUlWmi`Dvc+iY^F*O#gp?XGQG{QqOJTT{G*aNXobMP* z8;1r*_Eag+t+0WRV~?#~XS(effSwzC{w|I#jU#%!pQ;YWkJrB0d-^cICK$54qx1#@ zU9#kQvAn0Q#wU~2GWEEyPWV_~lSe%eF-Vv&@bm2+?*G*#d5;bBlFMO#=U(uo;6+qz zjYf?AB5Hv<2xMc-%@nKV=2pMx)<(RA-(R>}nZ?y=V3wHugj1Q*Z*iMsWd@ozH#VCT za7%%oJ0tr7xZzCVHP@P(Ft971w}z(9EN@3&<^v|B!P`F*RH@(CA;-EnvF}V$;Xvl+3|WmH(H0s&mRqFR9Rv|Xf+1+b&CM;uawpSiNofs z5ukaoSmt?n?-;Asa=wD>QI`%hOS`r&gS-4YKk=3(cDjXL`nl@LjWNvewXS&m1^4P` z>s+&o_U1CF6L0z(UklVD_$q+Q`)KXQW*|&V?RHV?_5@3SPEg?VW(fdKq|$3pv^*n8 z(YZj}268U?sa0JF+wAAHSRz{QN7bIsW_&uog|`%y{j%wBUwwVvs6!!+G{Z7A2p!~g zyeZ$}uxex`V^ow^&UK+D;3>+@4q+!|UT-jy1TXWEQu&Gj8lXj57)led@Pm8gZ6Tna zIXIwqMHl)bTDdLE?=Ex^Z&s<{A6w6m6ELa$YWDNx1!nr^yg-Z3n4o+9<4@%59rdN_ zOC=uXo5spcUR&!#Pj%@LxTdF8a94rEHBpmt1?^=^gjvd(&MYuyKax$kR%>s<>=u_9 zHc^QuQri@MPXf_AI#U^JgRFqCnNYe^6a(Z*KX5Bwt7ltI@dMATz$fuTC!z2eLq1o- z;kwliA}vftnMhu19y-v z{w}_9el!4YqDJzZ)90?F`qH{F7{|v_Q%v&kID_CZK46yHf|(4)O#dZYOJ#q~V8gTU z?eS{}fK*UJ3E%1O$Ma?cPU!Ojlii2DXWku%+}+ZP4{3jpA=@+kxGmE3hPHev{I;>k zLW${*ItMe)HxsWoBVB3$rN#C4hut32?@+N-eyM@dvBoOb4#!NCX}CRkmD=#4SZWoW zJ$$m!c;+>H+x`CKvAE<3f?s1A$4cYT5Y~PZZI)lExky0*7;p0w$WhR8c z98scM*0oZ|<4FO1FwKZf?nEY&^XJm83O3`O?d+~qrQQ|qeoT%zWD*QA8cB{o951I! z=`C*(ohB^Yg{ju9l~+Kg?} zLf>IQV07H}(yGKgnf@1`cYsG3JhX}tPlg16iE$&>_O4Vy++Olc(ES%e?BPLJt*XM@ zYu4OeeDQuZ9*=ml@(nlm)nH_X2Zj>wd#7~s^Ec%U!-ADtw0)Ki%AL#HaMjc=Z}KtUadA51e{~6+mo_HplH{%BqRHN#H0at!g*y(b7kg^>z%-7Xx;_#YUi+5U~P=zxQMbJSfE* zHx*?*1OUBT>vFiG0#b4Alu%nAiQWeBQ%l^?ma|FAr2RzG1!5JsN}Wa zvFu$8?3jqv`2+x=nr}fuf!U(gSV5;9y8HWWnxozzQeCE?kza_eui)!qd6IWv@ReABD*b?} zfHyLo=UhF$OrGCy{SYPMb%v)89R#^0(~$F{NKzlm&;$P34(Uce7B6%%Q-4xPJH%JD zgzcEA7t?StmuIx@h(EsXyRwQIeGfQ_4e^=2Ln$^49pFIM?*zO8FZW?9oCbsDV+9id zcv#@o8&sDmRXkRo=Ir#Gdei#AACoM`livqyI*jn{Sk+8EAiXuUnVEb*BHxFGtkr`G z{p$f6YJsCxB4x#=4C|*g8hC>c zM_jaW23!LU<5JA@jf)01T|T8NcAm3D4Az>q1n3PnTW;~Dag3D@ZrW6;vlynrKs5Nv zoX1B`Vp#iId(CKSdn;wEA3ztOpGco4u96A>qbKp!)ku;~Wu@tX0nXP}{D)~Jf!A7S z-3Y@z2{E|^ZOWG*0Y1YQQA3X0 zNCHAvwQisTl=^z(MGZ{4_nIS@A}-+-kv(3#-RPCC=Qtyso^-?$tSQm17XPG85U6H2 zX*Q}guJW*gr@(BpF`(9bG92dL8Aq%8!MwsMw&U}4q$6DsOGcdYGyV5Xdu6?}|0dlre*MKWS@?!46B z*A3sL4o%7E>O{NKHs+6JcYn9Z`_(rW`SdlqcB zQDDjB+Zc$HX==5-|3c>Ze{n5(c%b~be2w&5h1fp75sSPsLv0iO?El{{dc&;kOkKJ1pP*42F*`v^n$^5>W%|j7tL6I_1M*^{V6q#HJ&wP^B1V5 zs~_92e#DtPfiA}D%)?8w4s=-a-iL0}T$P|5aHGJ!*4H|@sJ2aqmxDz!93Z)&Bhe6j zZhGf;tJ_e`cNJ=NCsNo>&C9Z6!og4Ap{}QX2{m2%A>#(|((lg&LPLESfAyHicTMX* zFaF(15W52Ga$`CuLb-8PN|&*rB#X$uxX73Lw`E?#onl+PIm0b%EsSECiv>Zse*o~r zFI!zgmSO2MgfHOS4WpyJtcUXF?fh;o_Gfd8C#7Xv$<1x-A1%0!S-tM~2mBr@En^*S zwr7ceTB!f#K;J(artrx#y?j)1K{#6IXNy}XWfcXw-*`2G4&jVI=fvrMzcqwv`aH>bj|%s&TPp?b9FR=>!Dm@)i1XELH(2jEGc_L&frwjLozjG65@XfMQD3!>QnyNm0p)>vHN4-jqA_O zLVknrkC+X7`YW~&K}O;9^1Nxk9bg-fo1tk7G`hup$Vwyi|<7{&4mLh7tuIL0fJ= zhJ)YsQkPL;=6gV7 zJ5-tvXh%t2TKpatTKqQuFAOX(vn0WLs%R{Q)9 z-v=4LYk~{`eZK4bsLNNK7XSP=%^;DI1)Ykwy=7jf(}r~?YI(h7I|rc}V*dmGzny>u z3Ri{N`F$2X#jb9aw+6#BaYa@rOd^KL%X;`;68Bk;%vZQJ0)xyM}+% z8pU&ZSpGInB7EohKgu7Alr;Za_m7}6G=H0SCpz$N@~VMS#s4;2LEzfS|7nxrKM@u4 zSK$9aOOm{Xj(>&eAEXDT{%mRW7hwO%a2lsCaQxLV{-85EmEw`V;^9xSWP0A(_7|`G zlNnRb0;KK z^^-FB7J!CvKydCqAuFfqBOt6ObiC*`nC0%TK=GO5j}|8>ODX z%rIKub%yu;GrQkYX8IRXc0KCNq?I~Uk}O8Mo~bfx%H?1CFH}HBQ+rL2cC_cV_jTs$ zI!c4jY<~0{hIL99@@!vql2H^m)Qm*X%YCoq!>5x0$HSIB1{^zdt{HG2xIue_eA?K-i&6{uliv=dWJ*SdmkkAkkr!FT zqD6m244E^BaLCVy;cemr!gzu(-vLe!py(4^o|Jyz06{eL6wRJdyrX3?eVMTd$19`K)p>OZ;Qe=_j@*A@@e#Q;K^+gNB8K{ggA9Vh;q z_Wo6(Y$Hdza5np|Rfm6>EFa7NdSD!uEl}}j;}^S0ALia3&nN}E3-NHeCBW65UnCxL zD0fim&6rWfu^Y5TOoM?x%M=rWhhaiY95{7b@J?&$$5gmRFFM;=NMq%2R`sKNdxKdI zebm>5R(3gc2+wJ{S%6-eddLYw^Jjj}|I0HG-|9ms&*vp28rYbO@N6iS{nK)gvI;S3 zo8b?;ax(i@hW{^>HN?l^K$EnA4tsX+t0v*Xa>$fqDPJenh6#?}q3qD2(6ow8XU>f; zwDq7v0uPR|T8S+V#A@{2KlE-Obd`D_;LrdJx3$~JmY>f!a@Z;YZaZfhatMD@N(ld9 zslyQYpJWsG-;iwnkI{NsC5)1@#)@$6R2)({h+VTOEn}-}-i(Q$KFphqdnkEqM1|cm z^E$r6XrgWqAc`VfrHy*^QRdK;u`(6j^Mvezq6;QM4;{avR8kN~K1wP6|EiDuE|RU$ zm#)X1?O&-VEo%{H=DWt`A_1&hwwNA#y&tOXK-Mp1)?sEElpxNx2diY4L-IlF`j_Y0 zV7^|~%DnxU#!&!WzMF!3h2+L&;(j8ffp=+3${#mHfW5yO89>u0oz;D;PtQ_ApJclY za-VS;iQAwO83)a_HH?G>qkXX}_AKEjqM=M5uPvsLS#7b>YqZiTfV#D}vRT?Nu0+PS z@=Wu0NfVHN(8Tlo8v{^{)BV}GH?U{lTU4Lfp>DngbRaFIO;)CX$?lvOan(S5UiEx_ zU{uu&m`1x6Yb@o7M<35dM)6ibAflP0*hQQV(8gMDlPS5iDzf~lZKU4Y439t-g9+D5v9KbPvu*%qddyiSP1~>P@YM~ zY!sKJcUXdrEwQ#84dWIJVaV^@9L3_4JvSDeQ`gLT;|Y+iYZFtvJ=cCq0tBeA)5j6r zp#^keDBqG%>UXVI>AN&WW+@u)r8E?nAil6sPmX`UB@DvE!Z2H(|9ZrYWbrFgVR zPPb?H>v=+dUZ&AXbKgIqWx!y@peFI0A&@H$IN#7RMt95Oa zrr#QjcZKeoyPAnlGm?Chy2U!YLF`bAM#f9$*GlP3#oVem`^4G?G3=GE3SYf_bqhS_ zCEKsWNqL;_<5I@?+NX?~x~o(7IB(4A^0{XO{dfbkRnvf?W2E%ew9Abvt8cvDnkox2 zeB=A7JhD=65i=+fk!E(0ts~5H4E| z@AS=mHs_o zvLo3CbVk`gPzROD1F;VnF@>NH@H;dl*G4l?lWjN$V1TFI{68QYP@D}9x5GI!flmt4!zExi`$gtf9LD&N`c>(XZL)WF#5h(Q}7 z0QmL~P`lF3h*zEQ6up_G1l2HNW4pXXXdWs&1<~W$Vx5~Lr!MxMLQ{8K((3mvY5neq zIyv?f8?uS$E+qgFTG72#y$ksOHYP-G*l2{lwN+zDg@bg~{0Hw2@765X4bLwT8BJ+$ zegXLKW%7ffi4+n&QrDTkSJ`K5rk4TSK?x>e-AcOgh%v6fL0J{VaE`CSKQb>h=^ zYP+;YGCwe>{_eXcxRhfEW>LxNJ+#8vqn*;5q3$mXw1A{@BVWiI=+ZBmC(mM%0b#16 zBUf6n5{Z0P(}UHLq}HnOf{32$V=k@=z#I&`VVT#=NMqP|@)mTrA;J}|V0uFz8Xajc zJH3e7aK2Vx;D8+u&tu|v7Z0Ba6jGqOQn#LU@IWvkL7y#a0=^_$}2M(a(r)*JhmA&H)&sz zUm(9nFA7*21%Nx)Mzz|@Hap`bkeRRS2uXalzp{dV4E*X2kn2G5Y2MxlTk0HUm9%MH zsB5?J<7qhbXNG|Q$XzB;zZo(Ex!>Jnx?WiSuR1qZ2KMVyO_~!XMM%7nr2SK{BqFo?^R{f4tel{0*rrs5Ea>DV`g2F36vR)YWps zOt@o|N8}W$ExCATN+XT=>KH{rb2ZPm+8yhiw5Fc0_fhKW)^5C>)R4;S>17s?Ls+nH zopi@E*B55AumIUNi=SN<@$2=5U>Z^|YoYIGe3LAN!was^`(hf3>`NFzsQ~Lr`@0!B zewx_0{MC{ovTqge^B*> zom^*!OZYHdbg{?E#2M|l0ITmz{>n@HbWo zLDFJK^Yuh zQl1`?e!4|(fjiXW3Xr=z_nH)CZ|qXu%c&0Gfz`60%Dh3@)J>)F-HZ+UmYl_FK(}j- zLto|O!|~6(b79{EA+O;zM==zkwZr&JY4OPxIRa`)d~7mmVGG{Zn&acF($#`s`UUiE zsMn1EdRxanmKi(JcwU+z_NC$Y8J7X!St<*Bvyi`myR8>S!O2`bCJjCfj;rcSs$iFn z3QWx*)an)L3F!~^=71)>b2W*HK^@xjYA--Y46D|_>L~_|TW^GaDuA2_-y~L7Ok_Ps z&houl!(9n46a(GhrzLf%|3Cr7ZQI$L9MDa>Uh~O>uSZPNho|gFU;NHbZq;%6$f@J1 z9>_1OUSj$YB)4$#ix$?PzX*_TAoRk11h8t?TkK*`K9b|L6u7%V$fLpukt%XWqZjyr za~kiyzMr!!&b>r9f!c`!oq;r2jWd)ocNgj~PF!I@dft9QdKN5`q<_&{UFE(QE>eFF zsf65`BkVLOIAzAxHD7b_9djKep`Oho){2utLK-eBI8DBh2IZOI*-%op;MbYo>`3>F z1}!s(sy9AwIC)=7%vr2$t{Bg=7Lsobe)FYdSbflSwZr_U6>g$ZDf`^<-BKD5r0JQ< z$PYHT10rK0v9Mh{tW#TYH8wvn|H(XRfgV4<@}?fGh4slMywTn`qgpt{CP4E#hj6^Q z>pC`B##cYHd874=Yba$XK%sXBV(9%@V#7W`Yif){>^#U-Bv6hdi40&9)CyA;cFgH^ zc<94M1-q_XIQ)b^z&)VhsumXY4qNEnF(d5n`}C{lnkT7~Nx^Os_ERp4_}*f`s*Zvk z&s;AN=2fH=v4HhC$vG=Ta^?nSnoY;oaMlbi=9$(m$bEM~Ch|OKc#-?E-X{X|aP+v_IiKVwDFRMjYLau@zb>CvqG+CyLul5N9mw$H=I)YNet`>Fj!1Wk z{4#DcIG^T`^r?R{-45`2zMO_f)kRZUl2p^KwNh69)ubS{RDHqZH<3`#HSsznXoQ}a zrXZ>zDJo-_nERk6Lomz(>3Q!5d+~X|xEKn_Tw~r;>D{Mgg51;HzN`Whl&L5Oi>^Hx zYR*+Az1=t5hEZ%FB^;;+bXhX2xe+;XvlG*IaiJTD|Wpq1Uec z=INRbsK%P#%>KU=N&c_VBB%69s4PzgU)cQ@v#ytbDlbYO{aX)j`hR#Llz%?}bu#vx zP8L9;*O?Q=6|0coL3yM$hAy@ja1c`egz`nN4Ws?e>Q4EaoZ}<0z*5lEcDug`L-XpG zUnCP{>C{a=KW(dz%KYv}S(@N3G@dTV$h36h1kWWs0~EqSB;`j+pBA79^bymQ7b0BS zr2);)u+p6jVa<8mRjq(9p*4jeoOGYJRK+H*%hy{9EWa8$*~7~(*gHrCD#k8YSWtgd zoicVXo@Ry(h=ql6GCF%V7fXXW&^BXGzmZV6riHZXRj}LHK2?xE_#$xQtVnjJy@zZV zn|{9Mb*6CW6n32d!BD%@1o%_w@o zbBrYwdfoacH5t``0dO&i!HQdg>%1i5mfF$U3Suzvs1RT7PP9mKDYHEpYTcn})8=LZ z@2CNAF&a7aXNtp_-d$yAj}NjEFq-R{zL*g~SsdPk+HYY>X5by#&VX3!xffTX@5QIW z(cd0Z9&mmaoD10JqjY#2bVgwJ*=E`7RUrOKHf^Y|3lk?(g$@)j$dq*%lxIG~O9Ujs z4Nf5wYmN@prsNF-P!=*N)p|>l*bnbjVc$}|OnZG1OY$oNA+rco_6-2~X+Mf}Xl;bN z=?ul1o_e)#b)m0*cOhi(M7`AD7LdJE7P)%MxI;T#EdhSD_Uf$zr0k2?tbcvhz`x$# z@n1i$;P5`kky<1w&W$aF;ECnNK(25X22#ZGHXeevAXBTI4_@3NPgeoIlfdG$q;52> z&z!WM+($gRuFSnj5+_tPDl0B-F+hTYxx!n68_8^Ul?<;1O;R1~ISVZBaPxQXjV@!w zkDDt)N=B3Y6jF^@cbRNm_MV)NT-mVPwt+NEC9Cc1+crYB*QJ8D>n4|b)=A0xs)=OZ z5#^ctTO{Q>o3>~(UGv;@=ll?GZLs!zQqR%1^g6c=R zoYM8GNr>OxuIwIW8KRIKn8x2DzI0thbQkKN^U_i&5UksKqJePqD=?;JD0fTVV5i^!$lqpJ9%(3OF4XMVOcA)3A%rtBRh>R zKZYT9i3b-jvLmh?$!RmU6-<82v?mPSv?IfXMwB1MiB#XsG#ydiB59dNpeC>RKsGa* zXqA?g=_Z>1!4-qK=?jR4bR`L(T~BaeEQme4m8D-%X_4nb^Du-2YB;R{bwA398}P~9 zGYj2Hfh(I1ZkLjWhPIVW->*3}lB*%br#^JVONnX?h^F$%zoh}?9w*gvxy^(KHv&NN z`pTl443o{Y2a7DyBKcB;*JHWW8A))Y%3a6s2JE@Cg6ale>$#JD5un|gxVaaAO@=s) z7&0bPJ_x(aV&3%nAq;L*8Qjt@1OseHhOVt^5d|$RI<|NH8p(R}qru7iKWxtsmiG@( zeKiu4Z|xKE&AlfW)8sFavOhVBn~Gys=h-s7!KkD0< z{-D1?!9tWWX+K9+*`Wk7Vy7vaAvLuuPKxlId4l74-qlFbQa<>;u|ua8Irp^lfTY(( z)esy!q8wN;bUp2Us{|@yKW@2klOdCbr%`dS(Xf#6HU~|2&t?+~m^3oxTHGtpCjf zE8ZP0^q|e0aW33$k}Pt!(_Z6au@h7!EUUzi1eM>h&v#FCQ+TdP&)a3Ot)*aUdcw;@P2wJnm8hK=7UgN_ zFz8qLQGO=dW%u>_vn8 z1B*p5)^l5`J#FIs%0@yFDm!mF-^sO+kWY?TU)CmuzP!Ld44(qZwJ#tOd!>T$e$psk z{eEtzTaZ&dAzo^RrK{Vu_u+&DNErKph?ID>_^OW&l6he;6QQqYwO=YqTxovJ)OWC{ z%p-c-;&pIPxlWsT$tg0o_QH^;uSw;IZdfQeFKIO{Y-yoSOByRH!ems#O|bSNBYL)U zJoh2&k29A%KP%wpgun;l!lVWu?#_lL`;9r!IaHUJ;knN|}fGZ0*P^2vKFp zTyo@A23)+t*a6AxaJyT3tZ9gvt2Y}Ktbi3tezPAtD8cM3dw#CedY`qt2ogt_H-pAs39WOv9W+$Lmxf}@y^AZAGj}tGqlwIj9wH6RDg(~6MA62Y0 zdABO>0tVfw)ohiw8Xd=Y!DWh3HxSNnqh(p0$!X4H#o9#QmJrYll#i zWI8V7vCDRB{qbZASCE;TJm{rJLeI1b(20)b0tOC=y6PD1`Q#14a}eALIBEHG@kFjK zs){`~yG>DJUCV>EE6oI#O>1xhr;f&8O02%ebvWj8Zz_JTH!F%2!Ns8wU8oPyX7yPK~VR%?Xc zD2I-&gM_7oDu5QNzIK4+)`XGNwn>ZPj`(!I=ng6K`piY3^7JBM1)A^AJ+GvXx26N-EpSM~z0RGoht4Ox%N$C3vv?I)&6%oa0Oj1taFKZ8 zU>$4lqfH7Uy64phDF7kK9+H;*jQ-OFU2+-5X`R}1?h28eEZ2cn!(gGxcr^^%bv6Fu zorCez80PgW6>LzQwRNy8==!6&FV>vaLTSM{OBMV;_*9XUvZVo~3dWi6JYBmj3sak* zKlP;c1PH0whw`4oShZe~@7B)v;akhhaTh$Nyz+4Ct)GvYsrO&B3VG7m8Z?hX^QXW`W2cR$b73NLpis`4g zBknM_QZ}PTN7&?6yS?qvCNzfvMpKjcZnSHgoP0mS_`D)hkJ3=FW)Fx7j(s&o{8G`g|~J|lqzNOc)!ja z)xqLl895Z7rt=FO&4v_0Ji#5#3M5_O$=3ij^8yR|QtPO7#qsek;Eo|c;F?64A2=$Q zv_^8<#+dLh1mvS!GMj`c2_WJMFqdapZ84#k5?(+;BTiiA(<=Kvc8TssM{Z~FAZi`c zUtYHEyAhiRZ{+s|46ZIozezIu?lImp6gU9zG6*1?n4a85H2^on#v>?)w*~68%#inq zlgQ-0WZN1*(y7+zFSfa5PrhQ?K1igYV_^CclUoKhDatI+JHEsH?p^?+&?}d-Mfs0=X1@ZG@{Q~UYXFKfOUXQbrqlG|@nIUk0Tq&1@5eqz6xssLV3)amXr+kE1IxQPn0_|_ei0x z$bG54YSJy$D!TpQzK2@DNg{eXXWV}%EVNguS)^<;H(0UOGM3+R~3S3?u<{+ z?7crBKVio>B#8i0>Y9P>`yJi#J!rni5sC!o{ibbh7O_e&)77kvuD5`C&M2*Y=lypt zWN1fnQ)itd`zYt+E{C|v$GNTB-b6e@UYVetqTdLT<$;A)u6Awz_pZkNjYs;bKts>c j?`c__YYHLnL(r$88v7s Date: Mon, 17 Aug 2026 09:52:08 -0700 Subject: [PATCH 4/7] Restructure nginx guide into QAT subdirectory Move the detailed NGINX QAT guide into software/nginx/QAT/, with its config files and test script under QAT/supporting_files/ and the results chart under QAT/images/. Rename the chart to nginx_qat_comparison_intel_amd.png to describe what it shows. Add a short software/nginx/README.md landing page that points to the QAT guide, leaving room for future non-QAT nginx entries. --- software/nginx/QAT/README.md | 267 ++++++++++++++++++ .../nginx_qat_comparison_intel_amd.png} | Bin .../supporting_files}/connection_test.sh | 0 .../supporting_files}/nginx_with_qat.conf | 0 .../supporting_files}/nginx_without_qat.conf | 0 software/nginx/README.md | 264 +---------------- 6 files changed, 274 insertions(+), 257 deletions(-) create mode 100644 software/nginx/QAT/README.md rename software/nginx/{image.png => QAT/images/nginx_qat_comparison_intel_amd.png} (100%) rename software/nginx/{ => QAT/supporting_files}/connection_test.sh (100%) rename software/nginx/{ => QAT/supporting_files}/nginx_with_qat.conf (100%) rename software/nginx/{ => QAT/supporting_files}/nginx_without_qat.conf (100%) diff --git a/software/nginx/QAT/README.md b/software/nginx/QAT/README.md new file mode 100644 index 0000000..b349db8 --- /dev/null +++ b/software/nginx/QAT/README.md @@ -0,0 +1,267 @@ +# NGINX with Intel® QuickAssist Technology (Intel® QAT) Optimization Guide +## Table of Contents + +- [Overview](#overview) +- [QAT Hardware Requirement](#qat-hardware-requirement) +- [QAT Software Requirement and Prerequisites](#qat-software-requirement-and-prerequisites) + - [Enabling the Required QAT Services](#enabling-the-required-qat-services) +- [async-mode-nginx Configuration](#async-mode-nginx-configuration) +- [Building and configuring async-mode-nginx](#building-and-configuring-async-mode-nginx) + - [Generating the Server Certificate](#generating-the-server-certificate) + - [Validating the Configuration](#validating-the-configuration) +- [Supporting Files](#supporting-files) +- [Benchmarking](#benchmarking) + - [Core Allocation and `worker_processes`](#core-allocation-and-worker_processes) +- [Results](#results) +- [Details](#details) +- [References](#references) + +## Overview + +Compression and cryptography take up a significant portion of resources in the data center. Hardware acceleration like Intel® QuickAssist Technology (Intel® QAT) can be used to offload the compression and encryption portions of a workload. Offloading these operations will free up CPU cores to do other work and will improve compression and cryptography performance. NGINX is the world's most popular webserver. It is free and open source software, distributed under the terms of a simplified 2-clause BSD-like license. The "Async Mode for NGINX" adds asynchronous capabilities to NGINX using the OpenSSL Async Infrastructure. + + +## QAT Hardware Requirement + +At least one Intel® QAT engine is required and the individual engine might need to be updated in the BIOS. The following steps should be performed to be ready to use the QAT device(s). + +1. Check for QAT device availability. This can be verified by running the following command: + +``` +echo `(lspci -d 8086:4940 && lspci -d 8086:4941 && lspci -d 8086:4942 && lspci -d 8086:4943 && lspci -d 8086:4944 && lspci -d 8086:4945 && lspci -d 8086:4946 && lspci -d 8086:4947) | wc -l` supported devices found. +``` + +The command reports how many supported devices were found. At least one is required. On the system used for this benchmarking, the output was: + +``` +8 supported devices found. +``` + +2. Verify that the QAT firmware is already loaded by using the following command: + +``` +ls /lib/firmware/{qat_4xxx,qat_402xx,qat_420xx}.bin* 2>/dev/null +ls /lib/firmware/{qat_4xxx,qat_402xx,qat_420xx}_mmp.bin* 2>/dev/null +``` + +The output of the above command should include 2 firmware files. Note that this can vary depending on the exact QAT device on your hardware. + +``` + /lib/firmware/qat_402xx.bin + /lib/firmware/qat_402xx_mmp.bin +``` + +If the firmware is not already available, it can be downloaded from the Linux kernel repository: +https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/intel/qat + +``` +cd ~ +wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_4xxx.bin +wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_4xxx_mmp.bin +wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_402xx.bin +wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_402xx_mmp.bin +wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_420xx.bin +wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_420xx_mmp.bin +sudo cp qat_4xxx*.bin qat_402xx*.bin qat_420xx*.bin /lib/firmware +rm qat_4xxx*.bin qat_402xx*.bin qat_420xx*.bin +``` + +After firmware is updated, the initramfs must be updated. This differs based on the Linux distribution. + +3. Verify that the kernel drivers are loaded using the following command. + +``` +lsmod | grep qat +``` + +The output should be similar to the following: + +``` +qat_4xxx 16384 0 +intel_qat 172032 1 qat_4xxx +``` + +If the kernel modules are not found, they can be installed using: + +``` +sudo modprobe intel_qat +sudo modprobe qat_4xxx +``` + +If the kernel modules could not be installed, it might be needed to either install them through a kernel configuration or to install them with the distribution's package manager. + +## QAT Software Requirement and Prerequisites + +The QAT driver is available either "in-tree" as part of a release kernel or can be built outside of the release. This document assumes the use of the in-tree driver that is already available with kernel after version 5.19. The distribution used for this benchmarking was Ubuntu 24.04 with the in-tree driver. + +QATLib provides user space libraries that allow QAT device access and expose APIs for use by higher level applications. The QATLib driver can be installed using your distribution's package manager. For Ubuntu 24.04: + +``` +sudo -E apt install -y libqat4 libqat-dev qatlib-service qatlib-examples libusdm-dev +``` + +QATzip is a user-space library built on top of the Intel® QuickAssist Technology (QAT) user-space library. It provides extended compression and decompression capabilities by offloading these operations to Intel® QAT Accelerators. + +``` +sudo -E apt install -y qatzip libqatzip3 +``` + +Depending on the use case, the user can configure the number of QAT engines to use with the workload. In "Managed Mode", the [QATLib](https://intel.github.io/quickassist/qatlib/index.html) library can be used to restrict the workload to a specific number of engines. + +Please note that "intel_iommu=on" will be required as a kernel parameter. + +### Enabling the Required QAT Services + +Each QAT device is configured by a `/etc/4xxx_dev*.conf` file, and the `ServicesEnabled` setting in the `[GENERAL]` section controls which acceleration services that device exposes. This setting must include the services your workload actually uses: + +| ServicesEnabled | Services available | +| --- | --- | +| `dc` | Compression/decompression only | +| `sym` | Symmetric crypto only | +| `asym` | Asymmetric crypto (public key) only | +| `sym;dc` | Symmetric crypto and compression | +| `asym;dc` | Asymmetric crypto and compression | + +This matters because the two optimizations in this guide use different services. The qatzip module (`ngx_http_qatzip_filter_module`) needs `dc`, while QATEngine handling TLS handshakes (`ngx_ssl_engine_qat_module`) needs the crypto services. A device left at the compression-only default will not accelerate TLS, and the CPS results below cannot be reproduced on it. + +Check the current setting: + +``` +grep -H ServicesEnabled /etc/4xxx_dev*.conf +``` + +To use both compression and TLS acceleration, set the following in each device's `[GENERAL]` section: + +``` +ServicesEnabled = asym;dc +``` + +Then restart the service and confirm the devices come back up: + +``` +sudo systemctl restart qat.service +sudo systemctl status qat.service +``` + +Note that the available `ServicesEnabled` combinations vary by QAT generation, and not all services can be enabled on a single device simultaneously. See the [QATLib Users Guide](https://intel.github.io/quickassist/qatlib/index.html) for the combinations supported by your hardware. + +## async-mode-nginx Configuration + +This optimization was tested with the following software versions: + +async_mode_nginx v1.0.0 +nginx 1.26.2 +OpenSSL 3.0.13 +QATEngine 2.0.0-1~noble1 + +QATEngine is the OpenSSL engine that `nginx_with_qat.conf` selects via `use_engine qatengine`. On Ubuntu 24.04 it can be installed with: + +``` +sudo -E apt install -y qatengine +``` + + +## Building and configuring async-mode-nginx + +[async-mode-nginx](https://github.com/intel/asynch_mode_nginx) can be built with: + +``` +./configure \ + --prefix=$NGINX_INSTALL_DIR \ + --with-http_ssl_module \ + --add-dynamic-module=modules/nginx_qatzip_module \ + --add-dynamic-module=modules/nginx_qat_module/ \ + --with-cc-opt="-DNGX_SECURE_MEM -I$OPENSSL_LIB/include -I$ICP_ROOT/quickassist/include -I$ICP_ROOT/quickassist/include/dc -I$QZ_ROOT/include -Wno-error=deprecated-declarations" \ + --with-ld-opt="-Wl,-rpath=$OPENSSL_LIB/lib64 -L$OPENSSL_LIB/lib64 -L$QZ_ROOT/src -lqatzip -lz" + +make +make install +``` + +### Generating the Server Certificate + +Both configuration files in `supporting_files/` expect a certificate and key at the paths below. These are not created by the build, so generate them before starting the server. The results in this guide used a 2048-bit RSA key (RSA2K): + +``` +sudo mkdir -p /usr/local/nginx_qat_module/certs +sudo openssl req -x509 -newkey rsa:2048 -nodes -days 365 \ + -keyout /usr/local/nginx_qat_module/certs/server.key \ + -out /usr/local/nginx_qat_module/certs/server.crt \ + -subj "/CN=localhost" +sudo chmod 600 /usr/local/nginx_qat_module/certs/server.key +``` + +This produces a self-signed certificate, which is appropriate for benchmarking but not for production use. + +### Validating the Configuration + +Before running a benchmark, confirm that the configuration parses and that any dynamic modules it loads are present: + +``` +$NGINX_INSTALL_DIR/sbin/nginx -t -c /path/to/nginx_with_qat.conf +``` + +A successful check reports: + +``` +nginx: configuration file /path/to/nginx_with_qat.conf test is successful +``` + +This step catches missing module paths, unreadable certificates, and syntax errors before they show up as a failed test run. + +## Supporting Files + +| File | Purpose | +| --- | --- | +| [`supporting_files/nginx_with_qat.conf`](supporting_files/nginx_with_qat.conf) | async-mode-nginx configuration with the QAT modules loaded and the QAT engine enabled. | +| [`supporting_files/nginx_without_qat.conf`](supporting_files/nginx_without_qat.conf) | Baseline configuration with the QAT modules commented out. Note that this still uses the `asynch` listen parameter, so it must be run with the async-mode-nginx binary rather than stock nginx. | +| [`supporting_files/connection_test.sh`](supporting_files/connection_test.sh) | Drives the Connections Per Second (CPS) handshake test using `openssl s_time`. | +| `images/nginx_qat_comparison_intel_amd.png` | CPS results chart. | + +## Benchmarking + +CPS is measured with `supporting_files/connection_test.sh`, which spawns 200 concurrent `openssl s_time` clients against the server for 10 seconds each and sums the per-client connection rates: + +``` +./supporting_files/connection_test.sh +``` + +To print the commands without running them: + +``` +./supporting_files/connection_test.sh --emulation +``` + +The client count, duration, port, and cipher are set in the USER INPUT block at the top of the script. Note that the script's default cipher (`AES128-SHA`) and the `ssl_protocols TLSv1.2` setting in both configuration files do not match the TLS 1.3 / ECDHE-X25519-RSA2K configuration shown in the results chart below; adjust both to reproduce those specific numbers. + +### Core Allocation and `worker_processes` + +Both configuration files set `worker_processes 48`, which is deliberately fewer than the cores available on the test system rather than all of them. + +This reflects the scenario the guide is intended to demonstrate. In a real deployment, a web tier rarely has an entire high-core-count server to itself — it shares the machine with application, caching, or database workloads. The question that matters is therefore not "what peak CPS can this server reach with every core dedicated to NGINX," but "how much TLS throughput can be delivered from a modest slice of the machine, leaving the rest for other work." + +Offloading handshake cryptography to the QAT devices is what makes that slice go further. Because the asymmetric crypto moves off the cores and onto dedicated accelerators, the same 48 workers sustain substantially higher connection rates than they do without QAT — so the cores that remain free are genuinely available to other tenants rather than being consumed by TLS overhead. + +## Results + +![NGINX TLS handshake CPS, C4D Turin vs C4 GNR with and without QAT](images/nginx_qat_comparison_intel_amd.png) + +Intel® QAT is only exposed on bare-metal cloud instances, so this comparison is run there rather than on virtualized shapes. The two C4 GNR bars are the same bare-metal Intel Xeon 6985P system (`c4-highmem-288-metal`) described under [Details](#details), both running `worker_processes 48`, with the QAT modules and QAT engine as the only variable between them — the "without QAT" bar corresponds to `nginx_without_qat.conf` and the "with QAT" bar to `nginx_with_qat.conf`. The C4D Turin instance is likewise bare metal, so the cross-platform comparison holds the provisioning model constant as well. + +## Details + +NGINX on GNR (c4-highmem-288-metal), bare metal: Intel(R) Xeon(R) 6985P, 144 cores, 500W TDP, HT On, Turbo On, NUMA 6, Total Memory 2232GB, microcode 0x1000380, 4 QAT engines, Ubuntu 24.04 LTS, 6.14.0-gcp. Test by Intel as of Oct 6, 2025, async_mode_nginx version 1.0.0, nginx 1.26.2, OpenSSL 3.0.13, QATEngine 2.0.0 + +NGINX on Turin (c4d-highmem-384-metal), bare metal: AMD(R) EPYC(R) 9B45, 192 cores, HT On, Turbo On, NUMA 2, Total Memory 3072GB, microcode 0xb002150, Ubuntu 24.04 LTS, 6.14.0-gcp. Test by Intel as of Oct 6, 2025, async_mode_nginx version 1.0.0, nginx 1.26.2, OpenSSL 3.0.13 + +Results may vary. + +## References + +asynch_mode_nginx: https://github.com/intel/asynch_mode_nginx + +QATLib Users Guide: https://intel.github.io/quickassist/qatlib/index.html + +QATzip: https://github.com/intel/QATzip + +QATEngine: https://github.com/intel/QAT_Engine diff --git a/software/nginx/image.png b/software/nginx/QAT/images/nginx_qat_comparison_intel_amd.png similarity index 100% rename from software/nginx/image.png rename to software/nginx/QAT/images/nginx_qat_comparison_intel_amd.png diff --git a/software/nginx/connection_test.sh b/software/nginx/QAT/supporting_files/connection_test.sh similarity index 100% rename from software/nginx/connection_test.sh rename to software/nginx/QAT/supporting_files/connection_test.sh diff --git a/software/nginx/nginx_with_qat.conf b/software/nginx/QAT/supporting_files/nginx_with_qat.conf similarity index 100% rename from software/nginx/nginx_with_qat.conf rename to software/nginx/QAT/supporting_files/nginx_with_qat.conf diff --git a/software/nginx/nginx_without_qat.conf b/software/nginx/QAT/supporting_files/nginx_without_qat.conf similarity index 100% rename from software/nginx/nginx_without_qat.conf rename to software/nginx/QAT/supporting_files/nginx_without_qat.conf diff --git a/software/nginx/README.md b/software/nginx/README.md index 8d85404..f21ef26 100644 --- a/software/nginx/README.md +++ b/software/nginx/README.md @@ -1,267 +1,17 @@ -# NGINX with Intel® QuickAssist Technology (Intel® QAT) Optimization Guide -## Table of Contents +# NGINX Optimization Guides -- [Overview](#overview) -- [QAT Hardware Requirement](#qat-hardware-requirement) -- [QAT Software Requirement and Prerequisites](#qat-software-requirement-and-prerequisites) - - [Enabling the Required QAT Services](#enabling-the-required-qat-services) -- [async-mode-nginx Configuration](#async-mode-nginx-configuration) -- [Building and configuring async-mode-nginx](#building-and-configuring-async-mode-nginx) - - [Generating the Server Certificate](#generating-the-server-certificate) - - [Validating the Configuration](#validating-the-configuration) -- [Files in This Directory](#files-in-this-directory) -- [Benchmarking](#benchmarking) - - [Core Allocation and `worker_processes`](#core-allocation-and-worker_processes) -- [Results](#results) -- [Details](#details) -- [References](#references) +This section contains optimization guides for [NGINX](https://nginx.org/) on Intel hardware. -## Overview +NGINX is the world's most popular webserver. It is free and open source software, distributed under the terms of a simplified 2-clause BSD-like license. Because a web tier spends much of its time on TLS handshakes and on compressing responses, it benefits from offloading that cryptography and compression work off the CPU cores and onto dedicated accelerators. -Compression and cryptography take up a significant portion of resources in the data center. Hardware acceleration like Intel® QuickAssist Technology (Intel® QAT) can be used to offload the compression and encryption portions of a workload. Offloading these operations will free up CPU cores to do other work and will improve compression and cryptography performance. NGINX is the world's most popular webserver. It is free and open source software, distributed under the terms of a simplified 2-clause BSD-like license. The "Async Mode for NGINX" adds asynchronous capabilities to NGINX using the OpenSSL Async Infrastructure. +## Available Guides - -## QAT Hardware Requirement - -At least one Intel® QAT engine is required and the individual engine might need to be updated in the BIOS. The following steps should be performed to be ready to use the QAT device(s). - -1. Check for QAT device availability. This can be verified by running the following command: - -``` -echo `(lspci -d 8086:4940 && lspci -d 8086:4941 && lspci -d 8086:4942 && lspci -d 8086:4943 && lspci -d 8086:4944 && lspci -d 8086:4945 && lspci -d 8086:4946 && lspci -d 8086:4947) | wc -l` supported devices found. -``` - -The command reports how many supported devices were found. At least one is required. On the system used for this benchmarking, the output was: - -``` -8 supported devices found. -``` - -2. Verify that the QAT firmware is already loaded by using the following command: - -``` -ls /lib/firmware/{qat_4xxx,qat_402xx,qat_420xx}.bin* 2>/dev/null -ls /lib/firmware/{qat_4xxx,qat_402xx,qat_420xx}_mmp.bin* 2>/dev/null -``` - -The output of the above command should include 2 firmware files. Note that this can vary depending on the exact QAT device on your hardware. - -``` - /lib/firmware/qat_402xx.bin - /lib/firmware/qat_402xx_mmp.bin -``` - -If the firmware is not already available, it can be downloaded from the Linux kernel repository: -https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/intel/qat - -``` -cd ~ -wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_4xxx.bin -wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_4xxx_mmp.bin -wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_402xx.bin -wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_402xx_mmp.bin -wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_420xx.bin -wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/qat/qat_420xx_mmp.bin -sudo cp qat_4xxx*.bin qat_402xx*.bin qat_420xx*.bin /lib/firmware -rm qat_4xxx*.bin qat_402xx*.bin qat_420xx*.bin -``` - -After firmware is updated, the initramfs must be updated. This differs based on the Linux distribution. - -3. Verify that the kernel drivers are loaded using the following command. - -``` -lsmod | grep qat -``` - -The output should be similar to the following: - -``` -qat_4xxx 16384 0 -intel_qat 172032 1 qat_4xxx -``` - -If the kernel modules are not found, they can be installed using: - -``` -sudo modprobe intel_qat -sudo modprobe qat_4xxx -``` - -If the kernel modules could not be installed, it might be needed to either install them through a kernel configuration or to install them with the distribution's package manager. - -## QAT Software Requirement and Prerequisites - -The QAT driver is available either "in-tree" as part of a release kernel or can be built outside of the release. This document assumes the use of the in-tree driver that is already available with kernel after version 5.19. The distribution used for this benchmarking was Ubuntu 24.04 with the in-tree driver. - -QATLib provides user space libraries that allow QAT device access and expose APIs for use by higher level applications. The QATLib driver can be installed using your distribution's package manager. For Ubuntu 24.04: - -``` -sudo -E apt install -y libqat4 libqat-dev qatlib-service qatlib-examples libusdm-dev -``` - -QATzip is a user-space library built on top of the Intel® QuickAssist Technology (QAT) user-space library. It provides extended compression and decompression capabilities by offloading these operations to Intel® QAT Accelerators. - -``` -sudo -E apt install -y qatzip libqatzip3 -``` - -Depending on the use case, the user can configure the number of QAT engines to use with the workload. In "Managed Mode", the [QATLib](https://intel.github.io/quickassist/qatlib/index.html) library can be used to restrict the workload to a specific number of engines. - -Please note that "intel_iommu=on" will be required as a kernel parameter. - -### Enabling the Required QAT Services - -Each QAT device is configured by a `/etc/4xxx_dev*.conf` file, and the `ServicesEnabled` setting in the `[GENERAL]` section controls which acceleration services that device exposes. This setting must include the services your workload actually uses: - -| ServicesEnabled | Services available | +| Guide | Description | | --- | --- | -| `dc` | Compression/decompression only | -| `sym` | Symmetric crypto only | -| `asym` | Asymmetric crypto (public key) only | -| `sym;dc` | Symmetric crypto and compression | -| `asym;dc` | Asymmetric crypto and compression | - -This matters because the two optimizations in this guide use different services. The qatzip module (`ngx_http_qatzip_filter_module`) needs `dc`, while QATEngine handling TLS handshakes (`ngx_ssl_engine_qat_module`) needs the crypto services. A device left at the compression-only default will not accelerate TLS, and the CPS results below cannot be reproduced on it. - -Check the current setting: - -``` -grep -H ServicesEnabled /etc/4xxx_dev*.conf -``` - -To use both compression and TLS acceleration, set the following in each device's `[GENERAL]` section: - -``` -ServicesEnabled = asym;dc -``` - -Then restart the service and confirm the devices come back up: - -``` -sudo systemctl restart qat.service -sudo systemctl status qat.service -``` - -Note that the available `ServicesEnabled` combinations vary by QAT generation, and not all services can be enabled on a single device simultaneously. See the [QATLib Users Guide](https://intel.github.io/quickassist/qatlib/index.html) for the combinations supported by your hardware. - -## async-mode-nginx Configuration - -This optimization was tested with the following software versions: - -async_mode_nginx v1.0.0 -nginx 1.26.2 -OpenSSL 3.0.13 -QATEngine 2.0.0-1~noble1 - -QATEngine is the OpenSSL engine that `nginx_with_qat.conf` selects via `use_engine qatengine`. On Ubuntu 24.04 it can be installed with: - -``` -sudo -E apt install -y qatengine -``` - - -## Building and configuring async-mode-nginx - -[async-mode-nginx](https://github.com/intel/asynch_mode_nginx) can be built with: - -``` -./configure \ - --prefix=$NGINX_INSTALL_DIR \ - --with-http_ssl_module \ - --add-dynamic-module=modules/nginx_qatzip_module \ - --add-dynamic-module=modules/nginx_qat_module/ \ - --with-cc-opt="-DNGX_SECURE_MEM -I$OPENSSL_LIB/include -I$ICP_ROOT/quickassist/include -I$ICP_ROOT/quickassist/include/dc -I$QZ_ROOT/include -Wno-error=deprecated-declarations" \ - --with-ld-opt="-Wl,-rpath=$OPENSSL_LIB/lib64 -L$OPENSSL_LIB/lib64 -L$QZ_ROOT/src -lqatzip -lz" - -make -make install -``` - -### Generating the Server Certificate - -Both configuration files in this directory expect a certificate and key at the paths below. These are not created by the build, so generate them before starting the server. The results in this guide used a 2048-bit RSA key (RSA2K): - -``` -sudo mkdir -p /usr/local/nginx_qat_module/certs -sudo openssl req -x509 -newkey rsa:2048 -nodes -days 365 \ - -keyout /usr/local/nginx_qat_module/certs/server.key \ - -out /usr/local/nginx_qat_module/certs/server.crt \ - -subj "/CN=localhost" -sudo chmod 600 /usr/local/nginx_qat_module/certs/server.key -``` - -This produces a self-signed certificate, which is appropriate for benchmarking but not for production use. - -### Validating the Configuration - -Before running a benchmark, confirm that the configuration parses and that any dynamic modules it loads are present: - -``` -$NGINX_INSTALL_DIR/sbin/nginx -t -c /path/to/nginx_with_qat.conf -``` - -A successful check reports: - -``` -nginx: configuration file /path/to/nginx_with_qat.conf test is successful -``` - -This step catches missing module paths, unreadable certificates, and syntax errors before they show up as a failed test run. - -## Files in This Directory - -| File | Purpose | -| --- | --- | -| `nginx_with_qat.conf` | async-mode-nginx configuration with the QAT modules loaded and the QAT engine enabled. | -| `nginx_without_qat.conf` | Baseline configuration with the QAT modules commented out. Note that this still uses the `asynch` listen parameter, so it must be run with the async-mode-nginx binary rather than stock nginx. | -| `connection_test.sh` | Drives the Connections Per Second (CPS) handshake test using `openssl s_time`. | -| `image.png` | CPS results chart. | - -## Benchmarking - -CPS is measured with `connection_test.sh`, which spawns 200 concurrent `openssl s_time` clients against the server for 10 seconds each and sums the per-client connection rates: - -``` -./connection_test.sh -``` - -To print the commands without running them: - -``` -./connection_test.sh --emulation -``` - -The client count, duration, port, and cipher are set in the USER INPUT block at the top of the script. Note that the script's default cipher (`AES128-SHA`) and the `ssl_protocols TLSv1.2` setting in both configuration files do not match the TLS 1.3 / ECDHE-X25519-RSA2K configuration shown in the results chart below; adjust both to reproduce those specific numbers. - -### Core Allocation and `worker_processes` - -Both configuration files set `worker_processes 48`, which is deliberately fewer than the cores available on the test system rather than all of them. - -This reflects the scenario the guide is intended to demonstrate. In a real deployment, a web tier rarely has an entire high-core-count server to itself — it shares the machine with application, caching, or database workloads. The question that matters is therefore not "what peak CPS can this server reach with every core dedicated to NGINX," but "how much TLS throughput can be delivered from a modest slice of the machine, leaving the rest for other work." - -Offloading handshake cryptography to the QAT devices is what makes that slice go further. Because the asymmetric crypto moves off the cores and onto dedicated accelerators, the same 48 workers sustain substantially higher connection rates than they do without QAT — so the cores that remain free are genuinely available to other tenants rather than being consumed by TLS overhead. - -## Results - -![NGINX TLS handshake CPS, C4D Turin vs C4 GNR with and without QAT](image.png) - -Intel® QAT is only exposed on bare-metal cloud instances, so this comparison is run there rather than on virtualized shapes. The two C4 GNR bars are the same bare-metal Intel Xeon 6985P system (`c4-highmem-288-metal`) described under [Details](#details), both running `worker_processes 48`, with the QAT modules and QAT engine as the only variable between them — the "without QAT" bar corresponds to `nginx_without_qat.conf` and the "with QAT" bar to `nginx_with_qat.conf`. The C4D Turin instance is likewise bare metal, so the cross-platform comparison holds the provisioning model constant as well. - -## Details - -NGINX on GNR (c4-highmem-288-metal), bare metal: Intel(R) Xeon(R) 6985P, 144 cores, 500W TDP, HT On, Turbo On, NUMA 6, Total Memory 2232GB, microcode 0x1000380, 4 QAT engines, Ubuntu 24.04 LTS, 6.14.0-gcp. Test by Intel as of Oct 6, 2025, async_mode_nginx version 1.0.0, nginx 1.26.2, OpenSSL 3.0.13, QATEngine 2.0.0 - -NGINX on Turin (c4d-highmem-384-metal), bare metal: AMD(R) EPYC(R) 9B45, 192 cores, HT On, Turbo On, NUMA 2, Total Memory 3072GB, microcode 0xb002150, Ubuntu 24.04 LTS, 6.14.0-gcp. Test by Intel as of Oct 6, 2025, async_mode_nginx version 1.0.0, nginx 1.26.2, OpenSSL 3.0.13 - -Results may vary. +| [NGINX with Intel® QAT](QAT/README.md) | Offload TLS handshake cryptography and compression to Intel® QuickAssist Technology (Intel® QAT) using async-mode-nginx, including hardware and software prerequisites, configuration, and Connections Per Second (CPS) benchmark results. | ## References asynch_mode_nginx: https://github.com/intel/asynch_mode_nginx -QATLib Users Guide: https://intel.github.io/quickassist/qatlib/index.html - -QATzip: https://github.com/intel/QATzip - -QATEngine: https://github.com/intel/QAT_Engine +NGINX: https://nginx.org/ From 1839d02953d70b315305d42529c49ddf1d727b59 Mon Sep 17 00:00:00 2001 From: "Sherman, Srikanth" Date: Wed, 12 Aug 2026 09:02:20 -0700 Subject: [PATCH 5/7] Address review feedback on QAT detection and CPS test script Check each QAT device ID independently in a loop rather than chaining lspci calls, so the reported device count does not depend on lspci's exit status and stray errors are suppressed. In connection_test.sh, initialize emulation to 0 so the flag check is safe when the flag is absent, drop the stray backslash in the date format so all three timestamps are written the same way, and replace the ps/grep polling loop with wait, which returns once the client processes this script started have exited and cannot match unrelated processes. Expand CPS on first use. --- software/nginx/QAT/README.md | 8 ++++++-- .../nginx/QAT/supporting_files/connection_test.sh | 11 +++++------ 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/software/nginx/QAT/README.md b/software/nginx/QAT/README.md index b349db8..c9042ca 100644 --- a/software/nginx/QAT/README.md +++ b/software/nginx/QAT/README.md @@ -28,7 +28,11 @@ At least one Intel® QAT engine is required and the individual engine might need 1. Check for QAT device availability. This can be verified by running the following command: ``` -echo `(lspci -d 8086:4940 && lspci -d 8086:4941 && lspci -d 8086:4942 && lspci -d 8086:4943 && lspci -d 8086:4944 && lspci -d 8086:4945 && lspci -d 8086:4946 && lspci -d 8086:4947) | wc -l` supported devices found. +count=0 +for id in 4940 4941 4942 4943 4944 4945 4946 4947; do + count=$((count + $(lspci -d 8086:$id 2>/dev/null | wc -l))) +done +echo "$count supported devices found." ``` The command reports how many supported devices were found. At least one is required. On the system used for this benchmarking, the output was: @@ -122,7 +126,7 @@ Each QAT device is configured by a `/etc/4xxx_dev*.conf` file, and the `Services | `sym;dc` | Symmetric crypto and compression | | `asym;dc` | Asymmetric crypto and compression | -This matters because the two optimizations in this guide use different services. The qatzip module (`ngx_http_qatzip_filter_module`) needs `dc`, while QATEngine handling TLS handshakes (`ngx_ssl_engine_qat_module`) needs the crypto services. A device left at the compression-only default will not accelerate TLS, and the CPS results below cannot be reproduced on it. +This matters because the two optimizations in this guide use different services. The qatzip module (`ngx_http_qatzip_filter_module`) needs `dc`, while QATEngine handling TLS handshakes (`ngx_ssl_engine_qat_module`) needs the crypto services. A device left at the compression-only default will not accelerate TLS, and the Connections Per Second (CPS) results below cannot be reproduced on it. Check the current setting: diff --git a/software/nginx/QAT/supporting_files/connection_test.sh b/software/nginx/QAT/supporting_files/connection_test.sh index dc619be..b54548f 100755 --- a/software/nginx/QAT/supporting_files/connection_test.sh +++ b/software/nginx/QAT/supporting_files/connection_test.sh @@ -25,6 +25,7 @@ if [[ -z $ip_address || $@ == *-h* || $@ == *--help* ]]; then fi # Check for emulation flag +emulation=0 if [[ $@ == **emulation** ]] then emulation=1 @@ -47,7 +48,7 @@ printf " Cipher: $cipher\n" rm -rf ./.test_* # Get starttime -starttime=$(date +\%s) +starttime=$(date +%s) # Kick off the tests after checking for emulation if [[ $emulation -eq 1 ]] @@ -65,11 +66,9 @@ fi waitstarttime=$(date +%s) # wait until all processes complete -# The bracketed first character keeps grep from matching its own command line. -while [ $(ps -ef | grep -c "[o]penssl s_time") != 0 ]; -do - sleep 1 -done +# The clients are child processes of this script, so wait returns once every +# one of them has exited. +wait sumTotal=$(cat ./.test_$(($port))* | awk '(/^[0-9]* connections in [0-9]* real/){ total += $1/$4 } END {print total}') printf "Connections per second: $sumTotal CPS\n" From a80678d8e231a2daf816ac904f0008363f17ea63 Mon Sep 17 00:00:00 2001 From: "Sherman, Srikanth" Date: Mon, 17 Aug 2026 09:42:37 -0700 Subject: [PATCH 6/7] Document kernel driver requirements and build environment variables Address the two remaining review comments on the QAT guide. Replace the vague "install them through a kernel configuration" sentence with concrete guidance: how to check which QAT drivers the running kernel was configured with, which CONFIG options the 4xxx-series devices need, the per-device minimum kernel versions, and a link to the QATlib System Requirements page that documents the full kernel, firmware, and boot parameter requirements. Add a Required Environment Variables subsection before the configure snippet describing what NGINX_INSTALL_DIR, OPENSSL_LIB, ICP_ROOT, and QZ_ROOT point to, with an example export block. Note that ICP_ROOT and QZ_ROOT are source trees rather than the package install paths used earlier, and that the lib64 link paths assume a source-installed OpenSSL. --- software/nginx/QAT/README.md | 38 ++++++++++++++++++++++++++++++++++-- 1 file changed, 36 insertions(+), 2 deletions(-) diff --git a/software/nginx/QAT/README.md b/software/nginx/QAT/README.md index c9042ca..7a1e746 100644 --- a/software/nginx/QAT/README.md +++ b/software/nginx/QAT/README.md @@ -7,6 +7,8 @@ - [Enabling the Required QAT Services](#enabling-the-required-qat-services) - [async-mode-nginx Configuration](#async-mode-nginx-configuration) - [Building and configuring async-mode-nginx](#building-and-configuring-async-mode-nginx) + - [Required Environment Variables](#required-environment-variables) + - [Building](#building) - [Generating the Server Certificate](#generating-the-server-certificate) - [Validating the Configuration](#validating-the-configuration) - [Supporting Files](#supporting-files) @@ -92,7 +94,15 @@ sudo modprobe intel_qat sudo modprobe qat_4xxx ``` -If the kernel modules could not be installed, it might be needed to either install them through a kernel configuration or to install them with the distribution's package manager. +If `modprobe` fails, the driver is most likely not built in the running kernel. Confirm which QAT drivers the kernel was configured with: + +``` +grep -i qat /boot/config-$(uname -r) +``` + +The device you are using needs its driver built either as a module (`=m`, loadable with `modprobe`) or built in (`=y`). For the 4xxx-series devices this guide targets, that is `CONFIG_CRYPTO_DEV_QAT` plus the device-specific option — `CONFIG_CRYPTO_DEV_QAT_4XXX`, `CONFIG_CRYPTO_DEV_QAT_402XX`, or `CONFIG_CRYPTO_DEV_QAT_420XX`. If the option is absent or set to `n`, the fix is a kernel that includes it: a newer distribution kernel, a vendor kernel, or a locally rebuilt kernel with the option enabled. + +Each device generation also has a minimum kernel version — v5.15.3+ for 4xxx, v6.4+ for 402xx, and v6.8+ for 420xx — so a kernel older than that will not have the driver regardless of configuration. The [QATlib System Requirements](https://intel.github.io/quickassist/qatlib/requirements.html) page documents the full kernel, firmware, and boot-parameter requirements, including the per-device minimum kernel versions and the `intel_iommu=on` boot parameter noted below. ## QAT Software Requirement and Prerequisites @@ -167,7 +177,31 @@ sudo -E apt install -y qatengine ## Building and configuring async-mode-nginx -[async-mode-nginx](https://github.com/intel/asynch_mode_nginx) can be built with: +### Required Environment Variables + +The `./configure` line below is the one [async-mode-nginx](https://github.com/intel/asynch_mode_nginx) documents, and it expects four variables to be exported in the shell first. Nothing sets them for you, and configure will fail on missing headers if they are unset: + +| Variable | Points to | +| --- | --- | +| `NGINX_INSTALL_DIR` | Where NGINX will be installed, used as the `--prefix`. The configuration files in `supporting_files/` assume `/usr/local/nginx_qat_module`. | +| `OPENSSL_LIB` | The OpenSSL installation to build against. Its headers are expected at `$OPENSSL_LIB/include` and its libraries at `$OPENSSL_LIB/lib64`. | +| `ICP_ROOT` | The root of the QAT user space source tree, providing the QAT API headers at `$ICP_ROOT/quickassist/include`. For [qatlib](https://github.com/intel/qatlib), this is the top of the cloned repository. | +| `QZ_ROOT` | The root of the [QATzip](https://github.com/intel/QATzip) source tree, providing `$QZ_ROOT/include` and the `libqatzip` build output in `$QZ_ROOT/src`. | + +The `ICP_ROOT` and `QZ_ROOT` paths are source trees, so building the NGINX modules requires qatlib and QATzip checked out and built from source even though the runtime libraries installed earlier came from packages. For example: + +``` +export NGINX_INSTALL_DIR=/usr/local/nginx_qat_module +export OPENSSL_LIB=/usr/local/ssl +export ICP_ROOT=$HOME/qatlib +export QZ_ROOT=$HOME/QATzip +``` + +Adjust these to match where you unpacked and built each component. The `lib64` in the link flags reflects a source-installed OpenSSL layout; if you point `OPENSSL_LIB` at a distribution-packaged OpenSSL instead, correct those paths to that distribution's library directory (`/usr/lib/x86_64-linux-gnu` on Ubuntu 24.04). + +### Building + +async-mode-nginx can then be built with: ``` ./configure \ From d244f645113686e5a9dd07d31ce15fbaf3353e8d Mon Sep 17 00:00:00 2001 From: "Sherman, Srikanth" Date: Mon, 17 Aug 2026 10:24:03 -0700 Subject: [PATCH 7/7] Added link to NGINX optimization landing page --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 037d62a..39030b2 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,8 @@ We aim to provide a dynamic resource where users can find the latest optimizatio - [Java](software/java/README.md) - [Kafka](software/kafka/README.md) - [MySQL & PostgreSQL](software/mysql-postgresql/README.md) + - [NGINX](software/nginx/README.md) + - [NGINX QAT](software/nginx/QAT/README.md) - [NumPy](software/numpy/README.md) - [R (Rlang / Rstats)](software/R/README.md) - [scikit-learn](software/scikit-learn/README.md)