This suite is meant to serve as an example of how SPDK can be encapsulated into docker container images. The example containers consist of SPDK NVMe-oF target sharing devices to another SPDK NVMe-oF application. Which serves as both initiator and target. Finally a traffic generator based on FIO issues I/O to the connected devices.
docker: We recommend version 20.10 and above because it supports cgroups v2 for customization of host resources like CPUs, memory, and block I/O.
docker-compose: We recommend using 1.29.2 version or newer.
kernel: Hugepages must be allocated prior running the containers and hugetlbfs mount must be available under /dev/hugepages. Also, tmpfs should be mounted under /dev/shm. Depending on the use-case, some kernel modules should be also loaded into the kernel prior running the containers.
proxy: If you are working behind firewall make sure dockerd is aware of the proxy. Please refer to: docker-proxy
To pass $http_proxy to docker-compose build use:
docker-compose build --build-arg PROXY=$http_proxy
docker-compose.yaml shows an example deployment of the storage containers based on SPDK.
Running docker-compose build creates 5 docker images:
- build_base
- storage-target
- proxy-container
- traffic-generator-nvme
- traffic-generator-virtio
The build_base image provides the core components required to containerize SPDK
applications. The fedora:35 image from the Fedora Container Registry is used and then SPDK is installed. SPDK is installed out of build_base/spdk.tar.gz provided.
See build_base folder for details on what's included in the final image.
Running docker-compose up creates 3 docker containers:
-- storage-target: Contains SPDK NVMe-oF target exposing single subsystem to
proxy-container based on malloc bdev.
-- proxy-container: Contains SPDK NVMe-oF target connecting to storage-target
and then exposing the same devices to traffic-generator-nvme using NVMe-oF and
to traffic-generator-virtio using Virtio.
-- traffic-generator-nvme: Contains FIO using SPDK plugin to connect to proxy-container
and runs a sample workload.
-- traffic-generator-virtio: Contains FIO using SPDK plugin to connect to proxy-container
and runs a sample workload.
Each container is connected to a separate "spdk" network which is created before
deploying the containers. See docker-compose.yaml for the network's detailed setup and ip assignment.
All the above boils down to:
cd docker
tar -czf build_base/spdk.tar.gz --exclude='docker/*' -C .. .
docker-compose build
docker-compose up
The storage-target and proxy-container can be started as services.
Allowing for multiple traffic generator containers to connect.
docker-compose up -d proxy-container
docker-compose run traffic-generator-nvme
docker-compose run traffic-generator-virtio
Environment variables to containers can be passed as shown in docs. For example extra arguments to fio can be passed as so:
docker-compose run -e FIO_ARGS="--minimal" traffic-generator-nvme
As each container includes SPDK installation it is possible to use rpc.py to examine the final setup. E.g.:
docker-compose exec storage-target rpc.py bdev_get_bdevs
docker-compose exec proxy-container rpc.py nvmf_get_subsystems
- If you run docker < 20.10 under distro which switched fully to cgroups2 (e.g. f33) make sure that /sys/fs/cgroup/systemd exists otherwise docker/build will simply fail.
- Each SPDK app inside the containers is limited to single, separate CPU.