Skip to content

How to run SPDK

Jump to bottom
Keith Lucas edited this page Feb 9, 2022 · 6 revisions
  1. I tested this primarily on Ubuntu. Installing this seems to get all the dependencies:
apt-get install -y make linux-libc-dev pkg-config \
        devscripts libaio-dev libc6-dev gcc meson \
	python3-pyelftools uuid-dev libssl-dev \
	libibverbs-dev libfuse-dev libiscsi-dev \
	zlib1g-dev libfdt-dev libpcap0.8-dev \
	libncurses-dev libcunit1-dev \
	build-essential nasm autoconf libtool automake
  1. Checkout my version of spdk:
git clone https://github.com/keithalucas/spdk
git checkout longhorn
  1. Configure and make:
cd spdk
git submodule update --init
./configure
make
  1. Run

Before you run spdk, you need to run ./scripts/setup.sh. Then you can type:

./build/bin/spdk_tgt

You can run multiple instances of SPDK on a single host by specifying different socket paths on the command line with -r. The default socket path is /var/tmp/spdk.sock. For example, this starts SPDK with a different socket:

./build/bin/spdk_tgt -r /var/tmp/spdk2.sock
  1. Demo

You need at least two nodes. One or more remote replica nodes and one local replica node. On the remote replica nodes.

  1. Get https://github.com/keithalucas/spdk_replica_start on each remote replica node.
  2. Each replica node on equinix metal has /dev/sda available so that code uses it. If /dev/sda isn't available, choose something else or make a file greater than 10GB on each node and change the code to use that.
  3. Compile build with go build
  4. Run ./build/bin/spdk_tgt on each replica node. I run it in screen.
  5. Run ./spdk_replica_start <ip of node>

On the local replica node:

  1. Get https://github.com/keithalucas/spdk_volume_start
  2. Each replica node on equinix metal has /dev/sda available so that code uses it. If /dev/sda isn't available, choose something else or make a file greater than 10GB on each node and change the code to use that.
  3. Compile build with go build
  4. Run ./build/bin/spdk_tgt on the local replica node. I run it in screen.
  5. Run ./spdk_replica_start <remote node ip 1> <remote node ip 2>
  6. Discover the NVMe-OF target with nvme discover -t tcp -a 127.0.0.1 -s 4420
  7. Connect the NVMe-OF target with nvme connect -t tcp -a 127.0.0.1 -s 4420 -n nqn.2021-12.io.longhorn.volume:demo
  8. Find the device file with nvme list
  9. Format and use the device file:
mkfs.ext4 /dev/nvme0n1
mount /dev/nvme0n1 /mnt
Clone this wiki locally