thread: send message for completion with no channels

When spdk_for_each_channel() was called on an io_device with no
channels, the completion function would be called immediately in the
context of the spdk_for_each_channel() caller.  This is different than
the normal completion when channels are present, which is always called
asynchronously via spdk_thread_send_msg(), causing problems for callers
that did not expect to get the completion callback before
for_each_channel returned.

In particular, this causes problems with the set_bdev_qos_limit_iops RPC
method, which holds a mutex around the spdk_for_each_channel() call and
also attempts to lock that mutex again within the completion callback.
(This is GitHub issue spdk#344.)

To avoid this problem and make the completion call consistent between
no-channels and channels-present cases, always send the completion via
spdk_thread_send_msg().

Change-Id: I9e5ec3592462c7b2ed682e665fe62ae3a5b5cc59
Signed-off-by: Daniel Verkamp <[email protected]>
Reviewed-on: https://review.gerrithub.io/417068
Tested-by: SPDK Automated Test System <[email protected]>
Reviewed-by: Shuhei Matsumoto <[email protected]>
Reviewed-by: Ben Walker <[email protected]>

lib/thread/thread.c

@@ -647,9 +647,7 @@ spdk_for_each_channel(void *io_device, spdk_channel_msg fn, void *ctx,
 
 	pthread_mutex_unlock(&g_devlist_mutex);
 
-	cpl(i, 0);
-
-	free(i);
+	spdk_thread_send_msg(i->orig_thread, _call_completion, i);
 }
 
 void