SPDK 20.07

Signed-off-by: Tomasz Zawadzki <[email protected]>
Change-Id: Ic67fcdbf2c54b17355de3b2a04893f275a0c745a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3600
Tested-by: SPDK CI Jenkins <[email protected]>
Reviewed-by: Jim Harris <[email protected]>
Reviewed-by: Shuhei Matsumoto <[email protected]>

dpdkbuild: build dpdk with meson+ninja

Makefile support in DPDK was deprecated and will be removed soon,
so switch to the officially supported way of building DPDK -
with meson and ninja. Two new tools. Basically, our Makefiles
will invoke meson+ninja for DPDK, no other SPDK components are
affected.

Apparently DPDK wanted to move away from an octopus-like config
system and the ideology behind meson configuration is simple now:
build everything by default. Some PMDs can be explicitly disabled
with meson command line, but all libraries (both static and shared
versions) and test apps are built unconditionally.

How long does it take to build minimal DPDK with meson? Too much.
On my machine half of the total build time is spent on libraries
we don't need at all. (I have some hacks up my sleeve to disable
building those libraries - see the subsequent patch.) As for the
official way of building a minimal DPDK, there was a patch [1]
on dpdk mailing list to introduce more specific configuration,
but it was rejected:

> We talked about this a few times in the past, and it was actually one
> of the design goals to _avoid_ replicating the octopus-like config
> system of the makefiles. That's because it makes the test matrix
> insanely complicated, not to mention the harm to user friendliness,
> among other things.
>
> If someone doesn't want to use a PMD, they can just avoid installing it
> - it's simple enough.
>
> Sorry, but from me it's a very strong NACK.

Let's not follow that direction, hack the DPDK build system instead.

As for advantages of meson+ninja over Makefiles? I can't find any.
It's another build system that does a lot for you with some functions,
magic options, and a built-in dependency system. It seems nice if you know
the syntax, but it's another component that you need to learn, debug,
and possibly find bugs in (there's a lot of github issues open for meson).
I would compare it to CMake.

As for changes in this patch: rather that explicitly disabling
PMDs we don't need, specify a list of PMDs we do need and disable
everything else found in ./dpdk/drivers/*. This way we won't have
to disable the new PMDs as they're added to DPDK.

Meson configuration also sets RTE_EAL_PMD_PATH #define to a valid directory
with built PMD shared libs. When it's set, DPDK dynamically loads all shared
libraries inside. The drivers there depend on DPDK shared libs and fail to
load in static SPDK builds, so we disable them altogether by unsetting
RTE_EAL_PMD_PATH in the meson-generated config file - just like
DPDK Makefiles did. EAL checks for RTE_EAL_PMD_PATH being empty and skips
loading any external PMDs then. We do it for both static and shared libs.
We specify all PMDs at build time for now, so there's just no need to load
them dynamically.

We have three more hacks in our submodule:
 * disable building dpdk apps by commenting-out a line in dpdk/meson.build
 * disable building unnecessary libs (build everything that spdk *may*
   need)
 * build isa-l compress pmd with `-L[...] -lisal`. DPDK expects to find
   libisal with pkg-config. We don't want to prepare a pkg-config file,
   so comment-out a failing check in another meson.build file and provide
   isa-l through CFLAGS and LDFLAGS.

We also need to make some changes to our test/external_code. First of
all, -ldpdk is no more. Meson build generates a pkg-config file with all
libs, but we'll switch to it in a separate patch - for now just specify
all -lrte_ libs one by one. -Wl,--no-as-needed has to be added to some
test cases, otherwise rte_mempool_ring isn't loaded. We don't use any
APIs from this library, it only has a static constructor that provides
a few callbacks used by rte_mempool_create(). Also, since DPDK now builds
both static and shared libraries, we need to add -Wl,-Bstatic to force
using static libswhere required. It's only needed for DPDK libs, but we
use it for SPDK libs as well since there's no harm.

As for performance:
$ ./configure --enable-debug --with-crypto --with-reduce
$ time make -j40 -C dpdkbuild all
with meson:
real    0m8.287s
user    1m7.983s
sys     0m10.548s

before, with the old DPDK makefiles:
real    0m20.232s
user    0m55.921s
sys     0m16.491s

The subsequent builds are much faster too:
$ time make -j40 -C dpdkbuild all
meson:
real    0m0.876s
user    0m0.663s
sys     0m0.217s

makefiles:
real    0m10.150s
user    0m11.740s
sys     0m6.772s

[1] http://inbox.dpdk.org/dev/[email protected]/

Change-Id: Ic65db563014100bafb12e61ee0530cc2ae64401d
Signed-off-by: Darek Stojaczyk <[email protected]>
Signed-off-by: Tomasz Zawadzki <[email protected]>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1440
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <[email protected]>
Reviewed-by: Ben Walker <[email protected]>
Reviewed-by: Jim Harris <[email protected]>

SPDK 20.04.1

Signed-off-by: Tomasz Zawadzki <[email protected]>
Change-Id: Iea381a0886e331c3ccd0ec35ef60263dfa553b3e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2615
Reviewed-by: Darek Stojaczyk <[email protected]>
Reviewed-by: Jim Harris <[email protected]>
Reviewed-by: Ben Walker <[email protected]>

SPDK 20.01.2

Signed-off-by: Tomasz Zawadzki <[email protected]>
Change-Id: Id838a19f76bd5e7f6c771a3a4f673e85e4a1f92b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2698
Reviewed-by: Darek Stojaczyk <[email protected]>
Reviewed-by: Jim Harris <[email protected]>
Reviewed-by: Ben Walker <[email protected]>

SPDK 20.04

Signed-off-by: Tomasz Zawadzki <[email protected]>
Change-Id: I0e905f35fb8407f181c5341da6886980e8e49b81
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2096
Reviewed-by: Seth Howell <[email protected]>
Reviewed-by: Ben Walker <[email protected]>
Reviewed-by: Aleksey Marchuk <[email protected]>
Reviewed-by: Darek Stojaczyk <[email protected]>
Tested-by: SPDK CI Jenkins <[email protected]>

bdev/nvme: use g_nvme_bdev_ctrlrs as the poll group io device

IO device "bdev_nvme_poll_groups" is not unregistered in the finish
module, so we need to unregister this io device, since the "nvme_if"
is staic variable in bdev_nvme.c, it's can't be accessed in common.c
so we use "g_nvme_bdev_ctrlrs" instead.

Change-Id: Id4675f5acc8f386609903497da5ca84bf8af3a15
Signed-off-by: Changpeng Liu <[email protected]>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2035
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <[email protected]>
Reviewed-by: Shuhei Matsumoto <[email protected]>
Reviewed-by: Seth Howell <[email protected]>
Reviewed-by: Aleksey Marchuk <[email protected]>

SPDK 20.01.1

Signed-off-by: Tomasz Zawadzki <[email protected]>
Change-Id: I519f07a157f361141d3c2d9f4cf49af646af0901
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1375
Tested-by: SPDK CI Jenkins <[email protected]>
Reviewed-by: Jim Harris <[email protected]>
Reviewed-by: Ben Walker <[email protected]>

bdev/nvme: fix crash when invoking apply_firmware RPC

When structure for output of json decoders in not initialized
spdk_json_decode_string may fail trying to free uninitialized
string.

This patch changes mallocs used to allocate context and structure
for output of decoder with calloc.

Fixes spdk#1151

Change-Id: I180b2ec52350b4ca90e7c318b4f2d13af554ec49
Signed-off-by: Tomasz Kulasek <[email protected]>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/483107
Reviewed-by: Alexey Marchuk <[email protected]>
Reviewed-by: Tomasz Zawadzki <[email protected]>
Reviewed-by: Shuhei Matsumoto <[email protected]>
Tested-by: SPDK CI Jenkins <[email protected]>
Community-CI: Broadcom SPDK FC-NVMe CI <[email protected]>

SPDK 19.10.1

Change-Id: I3aa4fadc32a381cfe0686650db8bf409f0834ae0
Signed-off-by: Tomasz Zawadzki <[email protected]>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478449
Community-CI: SPDK CI Jenkins <[email protected]>
Tested-by: SPDK CI Jenkins <[email protected]>
Reviewed-by: Ben Walker <[email protected]>
Reviewed-by: Jim Harris <[email protected]>
Reviewed-by: Shuhei Matsumoto <[email protected]>
Reviewed-by: Alexey Marchuk <[email protected]>