diff --git a/README.md b/README.md index 85a6848..5bb7ff1 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,9 @@ Each workload represents a real-world use case with clear specifications, refere - [**Fetch-by-Similarity**](./fetch-by-similarity/index.html). Private database queries using cosine similarity search over encrypted data. -- [**ML Inference**](./ml-inference/index.html). Privacy-preserving machine learning inference on encrypted inputs. Currently features MNIST digit classification, with other models to come. +- [**MNIST Inference**](./ml-inference/index.html). Privacy-preserving machine learning inference on encrypted inputs for MNIST digit classification. + +- [**CIFAR10 Inference**](./cifar10-inference/index.html). Privacy-preserving machine learning inference on encrypted inputs. for CIFAR-10 image classification, with more models to come. - [**Zn Multiplication**](./Zn-multiplication/index.html). Multiplication of two encrypted 64-bit encrypted integers. Other bit-sizes for the multiplicands will be added. diff --git a/cifar10-inference/Large.html b/cifar10-inference/Large.html new file mode 100644 index 0000000..3232c63 --- /dev/null +++ b/cifar10-inference/Large.html @@ -0,0 +1,95 @@ + + + + FHE Benchmarking Results - CIFAR10 Inference (Large) + + + + + + + + + +

FHE Benchmarking Results - CIFAR10 Inference (Large)

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Submitter Bandwidth Quality Timing (harness) Timing (server)
NameDateEnvR/LModelKeysInputResultAccuracyTotalDB GenKeygenModel PrepInput GenInput PrepInput EncComputeDecryptTotalDB GenKeygenModel PrepInput GenInput PrepInput EncComputeDecryptI/O
+ + + + \ No newline at end of file diff --git a/cifar10-inference/Medium.html b/cifar10-inference/Medium.html new file mode 100644 index 0000000..500b94d --- /dev/null +++ b/cifar10-inference/Medium.html @@ -0,0 +1,95 @@ + + + + FHE Benchmarking Results - CIFAR10 Inference (Medium) + + + + + + + + + +

FHE Benchmarking Results - CIFAR10 Inference (Medium)

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Submitter Bandwidth Quality Timing (harness) Timing (server)
NameDateEnvR/LModelKeysInputResultAccuracyTotalDB GenKeygenModel PrepInput GenInput PrepInput EncComputeDecryptTotalDB GenKeygenModel PrepInput GenInput PrepInput EncComputeDecryptI/O
+ + + + \ No newline at end of file diff --git a/cifar10-inference/Single.html b/cifar10-inference/Single.html new file mode 100644 index 0000000..13cf1c7 --- /dev/null +++ b/cifar10-inference/Single.html @@ -0,0 +1,95 @@ + + + + FHE Benchmarking Results - CIFAR10 Inference (Single) + + + + + + + + + +

FHE Benchmarking Results - CIFAR10 Inference (Single)

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Submitter Bandwidth Timing (harness) Timing (server)
NameDateEnvR/LModelKeysInputResultTotalDB GenKeygenModel PrepInput GenInput PrepInput EncComputeDecryptTotalDB GenKeygenModel PrepInput GenInput PrepInput EncComputeDecryptI/O
Reference2026‑08‑05 03:38:16CPULresnet2050.4G28M8M33.52m14.34s1.0766m6ms2.9962s2.7333ms533.5ms32.14m370.2ms
+ + + + \ No newline at end of file diff --git a/cifar10-inference/Small.html b/cifar10-inference/Small.html new file mode 100644 index 0000000..61a8b6f --- /dev/null +++ b/cifar10-inference/Small.html @@ -0,0 +1,95 @@ + + + + FHE Benchmarking Results - CIFAR10 Inference (Small) + + + + + + + + + +

FHE Benchmarking Results - CIFAR10 Inference (Small)

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Submitter Bandwidth Quality Timing (harness) Timing (server)
NameDateEnvR/LModelKeysInputResultAccuracyTotalDB GenKeygenModel PrepInput GenInput PrepInput EncComputeDecryptTotalDB GenKeygenModel PrepInput GenInput PrepInput EncComputeDecryptI/O
+ + + + \ No newline at end of file diff --git a/cifar10-inference/index.html b/cifar10-inference/index.html new file mode 100644 index 0000000..c50a313 --- /dev/null +++ b/cifar10-inference/index.html @@ -0,0 +1,229 @@ + + + + +FHE Benchmarking: CIFAR-10 ML-Inference Workload + + + + + +
+ Repository is found at + github.com/fhe-benchmarking/ml-inference +
+ +

FHE Benchmarking: CIFAR-10 ML Inference Workload

+ +

Results - CIFAR-10

+ + +

Specification

+

The ML inference workload implements an encrypted inference functionality. +The input is a collection of 32x32 images with 3 RGB channels from the CIFAR-10 dataset representing 10 object classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck). +The goal is to classify these images into one of the 10 classes using encrypted ML Inference using homomorphic encryption. +

+ +

The workload includes two interfaces for benchmark submitters to implement:

+ + + + + + + + + + + + + + + + + + + + + + + + +
SizeRecords (N)
Small100
Medium1,000
Large10,000
+ +

Hence, there are a total of four variants of this workload: single inference and batch inference for each one of the three sizes. +Submitters need not implement all four, instead each submitter can implement and report the results of any subset.

+ +

Submission to the benchmarking suite must set the implementation parameters so as to achieve security level of at least 128 bits (against a semi-honest server). +Submitters must document their choice of parameters and explain why they believe that it meets the 128-bit security mandate. +(For example, for LWE-based schemes without a sparse key, they can rely on Table 5.2 or Table 5.3 in the HE-security-guidelines document of Bossuat et al. [BCC+24].)

+ +

Submissions are also required to meet the quality bar of correct inference result for single inference and at least 90% accuracy for batch inference. +

+ + +

The ml-inference harness contains a script that can be called to run the implementation of submitters, that script accepts command-line arguments to specify which interface of what instance size to run.

+ +
+

+$ python3 harness/run_submission.py -h
+usage: run_submission.py [-h] [--num_runs NUM_RUNS] [--seed SEED] [--count_only] [--remote]
+                         {0,1,2,3}
+
+Run the ml-inference FHE benchmark.
+
+positional arguments:
+  {0,1,2,3}            Instance size (0-toy/1-small/2-medium/3-large)
+
+options:
+  -h, --help           show this help message and exit
+  --num_runs NUM_RUNS  Number of times to run steps 4-9 (default: 1)
+  --seed SEED          Random seed for dataset and query generation
+  --count_only         Only count # of matches, do not return payloads
+  --remote             Run example submission in remote backend mode
+
+
+ +

Note that in the future, this workload will be updated to support more models and more datasets.

+ +You can find more details on the ml-inference Github repository. + + +
+

Bibliography

+
+

[BCC+24] +Security guidelines for implementing homomorphic encryption. +Jean-Philippe Bossuat, Rosario Cammarota, Ilaria Chillotti, Benjamin R. Curtis, +Wei Dai, Huijing Gong, Erin Hales, Duhyeong Kim, Bryan Kumara, Changmin Lee, +Xianhui Lu, Carsten Maple, Alberto Pedrouzo-Ulloa, Rachel Player, Yuriy Polyakov, +Luis Antonio Ruiz Lopez, Yongsoo Song, and Donggeon Yhee. +IACR Communications in Cryptology, 1(4):26, 2024. +

+
+ + \ No newline at end of file diff --git a/ml-inference/Large.html b/ml-inference/Large.html index f9692d7..df1dc83 100644 --- a/ml-inference/Large.html +++ b/ml-inference/Large.html @@ -1,7 +1,7 @@ - FHE Benchmarking Results - ML Inference (Large Batch - 10000) + FHE Benchmarking Results - MNIST Inference (Large Batch - 10000) diff --git a/ml-inference/Medium.html b/ml-inference/Medium.html index b60765c..c58ad18 100644 --- a/ml-inference/Medium.html +++ b/ml-inference/Medium.html @@ -27,7 +27,7 @@ -

FHE Benchmarking Results - ML Inference (Medium Batch - 1000)

+

FHE Benchmarking Results - MNIST Inference (Medium Batch - 1000)

diff --git a/ml-inference/Single.html b/ml-inference/Single.html index d8a2f70..b5c6ded 100644 --- a/ml-inference/Single.html +++ b/ml-inference/Single.html @@ -27,7 +27,7 @@ -

FHE Benchmarking Results - ML Inference (Single)

+

FHE Benchmarking Results - MNIST Inference (Single)

diff --git a/ml-inference/Small.html b/ml-inference/Small.html index bba8329..95cef13 100644 --- a/ml-inference/Small.html +++ b/ml-inference/Small.html @@ -27,7 +27,7 @@ -

FHE Benchmarking Results - ML Inference (Small Batch - 100)

+

FHE Benchmarking Results - MNIST Inference (Small Batch - 100)

diff --git a/ml-inference/index.html b/ml-inference/index.html index 10e32b0..516ee04 100644 --- a/ml-inference/index.html +++ b/ml-inference/index.html @@ -2,7 +2,7 @@ -FHE Benchmarking: ML-Inference Workload +FHE Benchmarking: MNIST-Inference Workload