Closed transports cause silent failures on writes

[sbonebrake]

Reproduced with Bunny 2.11.0 and RabbitMQ 3.7.6

If there is a closed transport, transport#write will silently fail when attempting to publish. This is when using automatically_recover: true and threaded: false.

Code to reproduce this issue can be found here: https://github.com/sbonebrake/bunny_repro/blob/master/heartbeat_failure_causes_silent_write_fails.rb

D, [2018-08-13T12:54:40.696182 #2936] DEBUG -- : Sent protocol preamble
D, [2018-08-13T12:54:40.701428 #2936] DEBUG -- : Sent connection.start-ok
D, [2018-08-13T12:54:40.702059 #2936] DEBUG -- : Heartbeat interval negotiation: client = 3, server = 60, result = 3
I, [2018-08-13T12:54:40.702093 #2936]  INFO -- : Heartbeat interval used (in seconds): 3
D, [2018-08-13T12:54:40.702128 #2936] DEBUG -- : Will use socket read timeout of 6.6000000000000005
D, [2018-08-13T12:54:40.702145 #2936] DEBUG -- : Initializing channel ID allocator with channel_max = 2047
D, [2018-08-13T12:54:40.702249 #2936] DEBUG -- : Sent connection.tune-ok with heartbeat interval = 3, frame_max = 131072, channel_max = 2047
D, [2018-08-13T12:54:40.702347 #2936] DEBUG -- : Sent connection.open with vhost = /
D, [2018-08-13T12:54:40.703017 #2936] DEBUG -- : Initializing heartbeat sender...
D, [2018-08-13T12:54:40.703150 #2936] DEBUG -- : Allocated channel id: 1
D, [2018-08-13T12:54:40.704355 #2936] DEBUG -- : Session#handle_frame on 1: #<AMQ::Protocol::Channel::OpenOk:0x00007f83ff2fb798 @channel_id="">
I, [2018-08-13T12:54:40.704409 #2936]  INFO -- : Declaring Bob with ch.queue('Bob')...
D, [2018-08-13T12:54:40.705170 #2936] DEBUG -- : Session#handle_frame on 1: #<AMQ::Protocol::Queue::DeclareOk:0x00007f83ff2fa848 @queue="Bob", @message_count=0, @consumer_count=0>
D, [2018-08-13T12:54:40.705237 #2936] DEBUG -- : Channel#handle_frame on channel 1: #<AMQ::Protocol::Queue::DeclareOk:0x00007f83ff2fa848 @queue="Bob", @message_count=0, @consumer_count=0>
D, [2018-08-13T12:54:41.105835 #2936] DEBUG -- : Sending a heartbeat, last activity time: 2018-08-13 12:54:40 -0700, interval (s): 0.4
D, [2018-08-13T12:54:41.511465 #2936] DEBUG -- : Sending a heartbeat, last activity time: 2018-08-13 12:54:40 -0700, interval (s): 0.4
# Use toxiproxy to down the connection
E, [2018-08-13T12:54:41.511860 #2936] ERROR -- : Error in the hearbeat sender: Broken pipe
I, [2018-08-13T12:54:43.708080 #2936]  INFO -- : Push some data to Bob
I, [2018-08-13T12:54:43.708585 #2936]  INFO -- : Silently fails because transport#write does nothing if not open

[michaelklishin]

Socket writes trigger connection recovery. Doing that and throwing an exception make little sense. More importantly, if you want reliable delivery, I/O exceptions is not what you are looking for: they might come way too late. Publisher confirms is what you are looking for.

[sbonebrake]

If the session doing a connection recovery, then it's status will be ":disconnected". When it is disconnected, calls to session#send_frame and session#send_frame_without_timeout will both raise ConnectionClosedError, which is good because then the client will know it failed. However, session#send_frameset and session#send_frameset_without_timeout don't perform the session#open? check. That means everything except basic_publish raises ConnectionClosedError. Why would basic_publish be different than all other writes operations?

[michaelklishin]

da6f972 provides an explanation.

[michaelklishin]

Bunny::Session#send_frameset_without_timeout could throw a ConnectionClosedError.