Fuzzing redisraft connection crash

[ds-testing-user]

We are a group of researchers testing distributed protocol implementations. While testing redisraft, we encountered the following crash (with a 3 node cluster) caused when polling for connections.

logs for node: 1

----- Stdout -----
811566:C 17 Oct 2023 18:45:55.951 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
811566:C 17 Oct 2023 18:45:55.951 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
811566:C 17 Oct 2023 18:45:55.951 * Redis version=255.255.255, bits=64, commit=e04ebdb8, modified=1, pid=811566, just started
811566:C 17 Oct 2023 18:45:55.951 * Configuration loaded
811566:M 17 Oct 2023 18:45:55.952 * monotonic clock: POSIX clock_gettime
811566:M 17 Oct 2023 18:45:55.952 * Running mode=standalone, port=5051.
811566:M 17 Oct 2023 18:45:55.957 * <raft> RedisRaft version 255.255.255 [c259f3d2]
811566:M 17 Oct 2023 18:45:55.961 * <raft> Initializing test network
811566:M 17 Oct 2023 18:45:55.961 * <raft> Created test network client
811566:M 17 Oct 2023 18:45:55.965 * <raft> Raft module loaded, state is 'loading'
811566:M 17 Oct 2023 18:45:55.965 * Module 'raft' loaded from /Fuzzing/redisraft-fuzzing/redisraft.so
811566:M 17 Oct 2023 18:45:55.966 * Server initialized
811566:M 17 Oct 2023 18:45:55.966 . The AOF directory appendonlydir doesn't exist
811566:M 17 Oct 2023 18:45:55.966 * Ready to accept connections tcp
811566:M 17 Oct 2023 18:45:55.966 . 0 clients connected (0 replicas), 1746720 bytes in use
811566:M 17 Oct 2023 18:45:56.008 - Accepted 127.0.0.1:58414
811566:M 17 Oct 2023 18:45:56.008 # Missing implement of connection type tls
811566:M 17 Oct 2023 18:45:56.065 * <raft> Loading: Redis loading complete, snapshot NOT LOADED
811566:M 17 Oct 2023 18:45:56.066 * <raft> Cluster Membership: term:0 index:0 nodes: id=1,voting=0,active=1,addr=-
811566:M 17 Oct 2023 18:45:56.066 * <raft> Test network starting message polling thread
811566:M 17 Oct 2023 18:45:56.067 * <raft> Cluster Membership: term:0 index:6 nodes: id=1,voting=1,active=1,addr=- id=2,voting=0,active=1,addr=localhost:5052
811566:M 17 Oct 2023 18:45:56.067 * <raft> Cluster Membership: term:0 index:6 nodes: id=1,voting=1,active=1,addr=- id=2,voting=0,active=1,addr=localhost:5052 id=3,voting=0,active=1,addr=localhost:5053
811566:M 17 Oct 2023 18:45:56.067 * <raft> Raft state after loading log: log_count=6, first_idx=1, current_idx=6, last_applied_idx=0
811566:M 17 Oct 2023 18:45:56.067 . <raft> Raft term=1, vote=-1
811566:M 17 Oct 2023 18:45:56.072 - Client closed connection id=8 addr=127.0.0.1:58414 laddr=127.0.0.1:5051 fd=11 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=0 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=auth user=default redir=-1 resp=2 lib-name= lib-ver=

----- Stderr -----




logs for node: 2

----- Stdout -----
811692:C 17 Oct 2023 18:45:56.136 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
811692:C 17 Oct 2023 18:45:56.136 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
811692:C 17 Oct 2023 18:45:56.136 * Redis version=255.255.255, bits=64, commit=e04ebdb8, modified=1, pid=811692, just started
811692:C 17 Oct 2023 18:45:56.136 * Configuration loaded
811692:M 17 Oct 2023 18:45:56.136 * monotonic clock: POSIX clock_gettime
811692:M 17 Oct 2023 18:45:56.136 * Running mode=standalone, port=5052.
811692:M 17 Oct 2023 18:45:56.141 * <raft> RedisRaft version 255.255.255 [c259f3d2]
811692:M 17 Oct 2023 18:45:56.145 * <raft> Initializing test network
811692:M 17 Oct 2023 18:45:56.146 * <raft> Created test network client
811692:M 17 Oct 2023 18:45:56.150 * <raft> Raft module loaded, state is 'loading'
811692:M 17 Oct 2023 18:45:56.150 * Module 'raft' loaded from /Fuzzing/redisraft-fuzzing/redisraft.so
811692:M 17 Oct 2023 18:45:56.150 * Server initialized
811692:M 17 Oct 2023 18:45:56.150 . The AOF directory appendonlydir doesn't exist
811692:M 17 Oct 2023 18:45:56.150 * Ready to accept connections tcp
811692:M 17 Oct 2023 18:45:56.151 . 0 clients connected (0 replicas), 1746816 bytes in use

----- Stderr -----




logs for node: 3

----- Stdout -----
811608:C 17 Oct 2023 18:45:56.012 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
811608:C 17 Oct 2023 18:45:56.012 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
811608:C 17 Oct 2023 18:45:56.012 * Redis version=255.255.255, bits=64, commit=e04ebdb8, modified=1, pid=811608, just started
811608:C 17 Oct 2023 18:45:56.012 * Configuration loaded
811608:M 17 Oct 2023 18:45:56.012 * monotonic clock: POSIX clock_gettime
811608:M 17 Oct 2023 18:45:56.013 * Running mode=standalone, port=5053.
811608:M 17 Oct 2023 18:45:56.017 * <raft> RedisRaft version 255.255.255 [c259f3d2]
811608:M 17 Oct 2023 18:45:56.021 * <raft> Initializing test network
811608:M 17 Oct 2023 18:45:56.022 * <raft> Created test network client
811608:M 17 Oct 2023 18:45:56.026 * <raft> Raft module loaded, state is 'loading'
811608:M 17 Oct 2023 18:45:56.026 * Module 'raft' loaded from /Fuzzing/redisraft-fuzzing/redisraft.so
811608:M 17 Oct 2023 18:45:56.027 * Server initialized
811608:M 17 Oct 2023 18:45:56.027 . The AOF directory appendonlydir doesn't exist
811608:M 17 Oct 2023 18:45:56.027 * Ready to accept connections tcp
811608:M 17 Oct 2023 18:45:56.027 . 0 clients connected (0 replicas), 1746688 bytes in use
811608:M 17 Oct 2023 18:45:56.068 - Accepted 127.0.0.1:60938
811608:M 17 Oct 2023 18:45:56.068 # Missing implement of connection type tls
811608:M 17 Oct 2023 18:45:56.127 * <raft> Loading: Redis loading complete, snapshot NOT LOADED
811608:M 17 Oct 2023 18:45:56.127 * <raft> Cluster Membership: term:0 index:0 nodes: id=3,voting=0,active=1,addr=-
811608:M 17 Oct 2023 18:45:56.128 * <raft> Test network starting message polling thread
811608:M 17 Oct 2023 18:45:56.128 * <raft> Cluster Membership: term:0 index:6 nodes: id=3,voting=0,active=1,addr=- id=1,voting=1,active=1,addr=:0
811608:M 17 Oct 2023 18:45:56.129 * <raft> Cluster Membership: term:0 index:6 nodes: id=3,voting=0,active=1,addr=- id=1,voting=1,active=1,addr=:0 id=2,voting=0,active=1,addr=localhost:5052
811608:M 17 Oct 2023 18:45:56.129 * <raft> Raft state after loading log: log_count=6, first_idx=1, current_idx=6, last_applied_idx=0
811608:M 17 Oct 2023 18:45:56.129 . <raft> Raft term=1, vote=-1


=== REDIS BUG REPORT START: Cut & paste starting from here ===
811608:M 17 Oct 2023 18:45:56.129 # Redis 255.255.255 crashed by signal: 11, si_code: 1
811608:M 17 Oct 2023 18:45:56.129 # Accessing address: 0x8
811608:M 17 Oct 2023 18:45:56.129 # Crashed running the instruction at: 0x7f6c40d7dffa

------ STACK TRACE ------
EIP:
/Fuzzing/redisraft-fuzzing/redisraft.so(ConnIsConnected+0xa)[0x7f6c40d7dffa]

Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f6c40fcf140]
/Fuzzing/redisraft-fuzzing/redisraft.so(ConnIsConnected+0xa)[0x7f6c40d7dffa]
/Fuzzing/redisraft-fuzzing/redisraft.so(HandleNodeStates+0x6b)[0x7f6c40d8397b]
/Fuzzing/redis/src/redis-server 0.0.0.0:5053(moduleTimerHandler+0x21c)[0x562abadc198c]
/Fuzzing/redis/src/redis-server 0.0.0.0:5053(+0x81705)[0x562abaccb705]
/Fuzzing/redis/src/redis-server 0.0.0.0:5053(aeMain+0x1c8)[0x562abacd5cb8]
/Fuzzing/redis/src/redis-server 0.0.0.0:5053(main+0x3cd)[0x562abaccae2d]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f6c40e0bd0a]
/Fuzzing/redis/src/redis-server 0.0.0.0:5053(_start+0x2a)[0x562abaccb5aa]

------ REGISTERS ------
811608:M 17 Oct 2023 18:45:56.154 # 
RAX:00000001ffffffff RBX:0000000000000000
RCX:00007f6c40fcb014 RDX:0000000000000000
RDI:0000000000000000 RSI:0000000000000000
RBP:0000000000000000 RSP:00007fff03bd7c78
R8 :0000000000000000 R9 :00007f6c40ddef80
R10:0000000000000000 R11:0000000000000246
R12:00007f6c40ddef00 R13:00007f6c40ddf4c8
R14:00007f6c4081ad00 R15:00007f6c4095dec0
RIP:00007f6c40d7dffa EFL:0000000000010287
CSGSFS:002b000000000033
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c87) -> 0000000000000040
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c86) -> 0000000000000000
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c85) -> 0000000000000000
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c84) -> 0000000000000000
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c83) -> 00007f6c4086f700
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c82) -> 00007f6c40829680
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c81) -> 0000562abada5cc0
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c80) -> 0000562abadc198c
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c7f) -> 00007fff03bd7d40
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c7e) -> 00007f6c4081ad00
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c7d) -> 000607ec41b20a78
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c7c) -> 00007fff03bd7cc0
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c7b) -> 0000000000000006
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c7a) -> 000607ec41b207fb
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c79) -> 00007f6c40ddef00
811608:M 17 Oct 2023 18:45:56.154 # (00007fff03bd7c78) -> 00007f6c40d8397b

------ INFO OUTPUT ------

----- Stderr -----

We've been able to reproduce the crash more than once.