feat: add storage and OCI lifecycle latency benchmark

[jim-junior]

This is the initial PR for submiting deliverables for Phase 2 — Midterm milestone of urunc-dev/urunc#759 thats adds the following:

• The harness program skeleton i.e the general structure on the harness program, This involes
• Benchmarking code for OCI Lifecycle latency
• Benchmarking code for Storage
• Jupyter Notebooks for analysis of benchmark results

Described in detail below

Harness Skeleton

The harness skeleton is a Go cli program. It works by taking in a experiment.yml manifest file, it validates it against the schema, generates an execution plan and then executes the benchmarks based. It also accompanies multiple helper commands ad demonstrated below.

$ ./harness --help
A tool to benchmark OCI rutimes and compare their performance. It provides a set of commands to run benchmarks, collect results, and generate reports.

Usage:
  evaluation_suite [flags]
  evaluation_suite [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  doctor      Check whether the host is ready to run benchmark experiments
  help        Help about any command
  plan        Generate a concrete trial plan from an experiment manifest
  run         Run benchmark trials from an experiment manifest
  validate    Validate an experiment manifest

Flags:
  -h, --help      help for evaluation_suite
  -v, --version   version for evaluation_suite

Use "evaluation_suite [command] --help" for more information about a command.

$  ./harness run --help
Run benchmark trials from an experiment manifest

Usage:
  evaluation_suite run [flags]

Flags:
      --dry-run              Use fake adapter and fake collector without running live commands
  -f, --file string          Path to experiment manifest
  -h, --help                 help for run
      --limit int            Limit number of trials to run
  -o, --output string        Output format: summary or json (default "summary")
      --results-dir string   Directory where run artifacts are written (default "results")

The experiment manifest is defined as follows in the example below(note that memory and CPU are not included)

runtimes:
  - name: runc
    handler: io.containerd.runc.v2

  - name: kata
    handler: io.containerd.kata.v2

  - name: runsc
    handler: io.containerd.runsc.v1

  - name: urunc
    handler: io.containerd.urunc.v2


experiments:

  lifecycle:
    workloads:
      default:
        image: docker.io/library/nginx:alpine

      other:
      - image: docker.io/library/minimalc:test
        runtime: urunc

  storage:
    workloads:
      default:
        image: docker.io/jimjuniorb/fio:0.1
        snapshotter: overlayfs
      other:

      - image: docker.io/jimjuniorb/fio-urunc:0.4
        runtime: urunc
        snapshotter: devmapper

OCI Lifecycle latency benchmark

For OCI lifecycle latency, the approach used for measuring is as follows. We listen to containerd runtime tasks for the rutimes runc, kata and urunc. For gVisor(runsc) we are running the runsc cli command with a bundle and other required arguments then measuring the execution time it takes, This is because runsc does not abide by the Runtime v2 spec.

Aditionaly for urunc, when measuring the start OCI command, we ignore the /tasks/start event because as mentioned in the previous weekly sync it publishes this task earlier than when the container starts. To solve this, we created a minimal C program that starts and exits immediately with 0 code. Its code is in workloads/minimal-c/main.c

Storage Benchmark

As for storage, we built custom images with fio included in them.

For runtimes kata, runc and runsc, we are using the image docker.io/jimjuniorb/fio:0.1

As for urunc we are using the docker.io/jimjuniorb/fio-urunc:0.4 image.

For urunc, the kernel included in the image harbor.nbfc.io/nubificus/urunc/linux-kernel-qemu:v6.14 was misssing certain system calls that fio depends on, this required me to build a custom kernel that would be used in this benchmark. To do this i adjusted the kernel config @cmainas used at https://gist.github.com/cmainas/223e1525496dd2c8e08dbf8bab41df80 and added support for shmget syscall by enabling the CONFIG_SYSVIPC and its dependencies. The final kernel build config file can be found in workloads/fio-kernel/.config

Analysis Notebooks

For result analysis, i opted to use Jupyter notebooks. I find them simpler and more powerful to use for analysis. Currently there are only two notebooks, fio_runtime_analysis.ipynb (for storage) and oci_lifecycle_latency_analysis.ipynb located in the notebooks directory.

For now the benchmark results(which are stored in a run.json file) should be copied and put in the same directory as the notebooks when doing analysis.

These notebooks generate reports and easy to read graphs.

Sample results

---

LLM usage

Used LLMs mainly to assist on the process/pipeline of how to build a custom linux kernel

Checklist

• I have read the contribution guide.
• If LLMs were used: I have read the llm policy.