Core dump on startup [0.8.0] #193

kallerosenbaum · 2019-11-14T21:23:18Z

I just upgraded from 0.7.1 to 0.8.0. 0.7.1 worked fine but 0.8.0 prints this on the log:

$ cargo run --release
    Finished release [optimized] target(s) in 0.87s
     Running `target/release/electrs`
Config { log: StdErrLog { verbosity: Debug, quiet: false, timestamp: Off, modules: [], writer: "stderr", color_choice: Auto }, network_type: bitcoin, db_path: "/mnt/bitcoin/.electrs/db/mainnet", daemon_dir: "/mnt/bitcoin/.bitcoin", daemon_rpc_addr: V4(127.0.0.1:8332), cookie: Some("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"), electrum_rpc_addr: V4(192.168.1.3:50001), monitoring_addr: V4(127.0.0.1:4224), jsonrpc_import: false, index_batch_size: 100, bulk_index_threads: 4, tx_cache_size: 10485760, txid_limit: 100, server_banner: "Welcome to electrs 0.8.0 (Electrum Rust Server)!", blocktxids_cache_size: 10485760 }
DEBUG - Server listening on 127.0.0.1:4224
DEBUG - Running accept thread
INFO - NetworkInfo { version: 180100, subversion: "/Satoshi:0.18.1/" }
INFO - BlockchainInfo { chain: "main", blocks: 603796, headers: 603796, bestblockhash: "00000000000000000015fd5c7fc95b2040c3975f8d4ee86dd9b1995a387f6cde", pruned: false, initialblockdownload: false }
DEBUG - opening DB at "/mnt/bitcoin/.electrs/db/mainnet"
Illegal instruction (core dumped)

I upgraded by

Stop electrs 0.7.1
git fetch
git checkout Cargo.lock (I had to in order to checkout v0.8.0)
git checkout v0.8.0
cargo build --release

Here's my ~/.electrs/config.toml:

cookie = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
verbose = 3
db_dir = "/mnt/bitcoin/.electrs/db"
electrum_rpc_addr = "192.168.1.3:50001"

I also tried to move away the db dir to start from scratch, but it didn't help.

The text was updated successfully, but these errors were encountered:

kallerosenbaum · 2019-11-14T21:32:14Z

$ rustc --version
rustc 1.38.0 (625451e37 2019-09-23)
$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
$ uname -a
Linux wilfrid 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

romanz · 2019-11-15T12:16:25Z

Thanks for opening the issue!
Could you please update to Rust 1.39 to see if the problem reproduces?

kallerosenbaum · 2019-11-19T07:26:56Z

Same problem with rust 1.39:

$ git status
HEAD detached at v0.8.0
nothing to commit, working tree clean

$ rustc --version
rustc 1.39.0 (4560ea788 2019-11-04)

$ cargo build --release
   Compiling libc v0.2.65
   Compiling cfg-if v0.1.10
...
   Compiling librocksdb-sys v6.1.3
   Compiling rocksdb v0.12.4
    Finished release [optimized] target(s) in 12m 29s

$ cargo run --release
    Finished release [optimized] target(s) in 1.96s
     Running `target/release/electrs`
Config { log: StdErrLog { verbosity: Debug, quiet: false, timestamp: Off, modules: [], writer: "stderr", color_choice: Auto }, network_type: bitcoin, db_path: "/mnt/bitcoin/.electrs/db/mainnet", daemon_dir: "/mnt/bitcoin/.bitcoin", daemon_rpc_addr: V4(127.0.0.1:8332), cookie: Some("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"), electrum_rpc_addr: V4(192.168.1.3:50001), monitoring_addr: V4(127.0.0.1:4224), jsonrpc_import: false, index_batch_size: 100, bulk_index_threads: 4, tx_cache_size: 10485760, txid_limit: 100, server_banner: "Welcome to electrs 0.8.0 (Electrum Rust Server)!", blocktxids_cache_size: 10485760 }
DEBUG - Server listening on 127.0.0.1:4224
DEBUG - Running accept thread
INFO - NetworkInfo { version: 180100, subversion: "/Satoshi:0.18.1/" }
INFO - BlockchainInfo { chain: "main", blocks: 604459, headers: 604459, bestblockhash: "00000000000000000003e246dc64bd4bd94b62d88ae984ec66111d1441c91d09", pruned: false, initialblockdownload: false }
DEBUG - opening DB at "/mnt/bitcoin/.electrs/db/mainnet"
Illegal instruction (core dumped)

romanz · 2019-11-19T12:21:06Z

Not sure but maybe it's related to rust-rocksdb crate #159 (comment)?

romanz · 2019-11-19T13:08:59Z

Could you please try 48622f4?

kallerosenbaum · 2019-11-19T17:16:24Z

Still same issue

$ git status
HEAD detached at 48622f4
$ cargo run --release
    Finished release [optimized] target(s) in 2.61s
     Running `target/release/electrs`
Config { log: StdErrLog { verbosity: Debug, quiet: false, timestamp: Off, modules: [], writer: "stderr", color_choice: Auto }, network_type: bitcoin, db_path: "/mnt/bitcoin/.electrs/db/mainnet", daemon_dir: "/mnt/bitcoin/.bitcoin", daemon_rpc_addr: V4(127.0.0.1:8332), electrum_rpc_addr: V4(192.168.1.3:50001), monitoring_addr: V4(127.0.0.1:4224), jsonrpc_import: false, index_batch_size: 100, bulk_index_threads: 4, tx_cache_size: 10485760, txid_limit: 100, server_banner: "Welcome to electrs 0.8.0 (Electrum Rust Server)!", blocktxids_cache_size: 10485760 }
DEBUG - Server listening on 127.0.0.1:4224
DEBUG - Running accept thread
INFO - NetworkInfo { version: 180100, subversion: "/Satoshi:0.18.1/" }
INFO - BlockchainInfo { chain: "main", blocks: 604511, headers: 604511, bestblockhash: "00000000000000000003c243c467b34fbc3140d4c7a6782d871e77dbb12dd5dc", pruned: false, initialblockdownload: false }
DEBUG - opening DB at "/mnt/bitcoin/.electrs/db/mainnet"
Illegal instruction (core dumped)

kallerosenbaum · 2019-11-19T18:40:34Z

I also cloned the repo clean, checked out 0.8.0, built and run with the exact same result.

romanz · 2019-11-19T18:47:12Z

Could you please try to git bisect the history between 0.7.1 and 0.8.0?
I think that running cargo test should fail as well.
BTW, which OS & machine are you using?

kallerosenbaum · 2019-11-19T18:54:34Z

#193 (comment)

romanz · 2019-11-19T20:08:39Z

Is it a cloud-based VM by chance?
If so, I can try to reproduce the issue...

romanz · 2019-11-19T20:14:27Z

Could you please try to git bisect the history between 0.7.1 and 0.8.0?

You can also try to downgrade just rocksdb crate version back to 0.12.1 since I think it was used at 0.7.1:

diff --git a/Cargo.toml b/Cargo.toml
index a73a731..671fc63 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "electrs"
-version = "0.7.1"
+version = "0.8.0"
 authors = ["Roman Zeyde <[email protected]>"]
 description = "An efficient re-implementation of Electrum Server in Rust"
 license = "MIT"
@@ -10,20 +10,23 @@ keywords = ["bitcoin", "electrum", "server", "index", "database"]
 documentation = "https://docs.rs/electrs/"
 readme = "README.md"
 edition = "2018"
+build = "build.rs"
+
+[package.metadata.configure_me]
+spec = "config_spec.toml"
 
 [profile.release]
 lto = true
 
 [features]
-latest_rust = []  # use latest Rust features (otherwise, support Rust 1.32)
+latest_rust = []  # use latest Rust features (otherwise, support Rust 1.34)
 
 [dependencies]
-arrayref = "0.3"
 base64 = "0.10"
 bincode = "1.0"
-bitcoin = "0.18"
-bitcoin_hashes = "0.3"
-clap = "2.31"
+bitcoin = { version = "0.21", features = ["use-serde"] }
+bitcoin_hashes = "0.7.1"
+configure_me = "0.3.3"
 crossbeam-channel = "0.3"
 dirs = "1.0"
 error-chain = "0.12"
@@ -35,7 +38,7 @@ lru = "0.1"
 num_cpus = "1.0"
 page_size = "0.4"
 prometheus = "0.5"
-rocksdb = "0.12.1"
+rocksdb = "0.12"
 rust-crypto = "0.2"
 serde = "1.0"
 serde_derive = "1.0"
@@ -45,3 +48,6 @@ stderrlog = "0.4.1"
 sysconf = ">=0.3.4"
 time = "0.1"
 tiny_http = "0.6"
+
+[build-dependencies]
+configure_me_codegen = "0.3.12"

kallerosenbaum · 2019-11-20T15:28:47Z

No, it's a physical machine in my basement. Nothing cloudy there, but a bit dusty.

romanz · 2019-11-20T18:29:58Z

Could you please build and run latest rust-rocksdb tests?

$ git clone https://github.com/rust-rocksdb/rust-rocksdb
$ git submodule update --init --recursive
$ cargo test

I am suspecting that https://github.com/rust-rocksdb/rust-rocksdb/blob/3c810163091fb8d343a3fa8373edd4b5685ed84d/librocksdb-sys/build.rs#L100 is causing the crash :(

kallerosenbaum · 2019-11-20T20:58:09Z

Right! Here's my output.

$ cargo test
    Updating crates.io index
  Downloaded cc v1.0.47
  Downloaded cexpr v0.3.6
  Downloaded num_cpus v1.11.1
  Downloaded jobserver v0.1.17
   Compiling libc v0.2.65
   Compiling log v0.4.8
...
   Compiling which v2.0.1
   Compiling librocksdb-sys v6.2.4 (/mnt/bitcoin/git/rust-rocksdb/librocksdb-sys)
warning: couldn't execute `llvm-config --prefix` (error: No such file or directory (os error 2))
warning: set the LLVM_CONFIG_PATH environment variable to a valid `llvm-config` executable
warning: snappy/snappy-sinksource.cc: In member function ‘virtual char* snappy::Sink::GetAppendBuffer(size_t, char*)’:
...
warning: snappy/snappy-sinksource.cc:39:36: warning: un
warning: bzip2/compress.c:256:11: note: ‘cost[2]’ was declared here
warning:     UInt16 cost[BZ_N_GROUPS];
warning:            ^~~~
   Compiling rocksdb v0.13.0 (/mnt/bitcoin/git/rust-rocksdb)
    Finished dev [unoptimized + debuginfo] target(s) in 8m 54s
     Running target/debug/deps/rocksdb-23fdbe4138dbea96
error: process didn't exit successfully: `/mnt/bitcoin/git/rust-rocksdb/target/debug/deps/rocksdb-23fdbe4138dbea96` (signal: 4, SIGILL: illegal instruction)

My CPU is pretty old:

$ cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
stepping	: 11
microcode	: 0xba
cpu MHz		: 1622.013
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm pti tpr_shadow vnmi flexpriority dtherm
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4800.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

... And 3 more cores

kallerosenbaum · 2019-11-20T21:02:22Z

My CPU (Kentsfield) is from somewhere 2006-2009 which is older than Intel Nehalem (about 2010)

kallerosenbaum · 2019-11-20T22:01:25Z

My options seems to be

a) Use electrs 0.7.1
b) Use electrs 0.8.0 but use rocksdb 0.12.2 instead of 0.12.4 as git diff v0.7.1 v0.8.0 Cargo.lock suggests.

kallerosenbaum · 2019-11-20T22:34:08Z

cargo update -p rocksdb --precise 0.12.2 did the trick! It works now, but what something else will stop working?

romanz · 2019-11-25T07:17:13Z

I think that RocksDB is the only non-Rust crate, so the fix above should work well.

romanz · 2019-11-25T07:24:14Z

FTR, this is an open issue at rust-rocksdb/rust-rocksdb#327.

dmp1ce · 2019-11-29T20:55:09Z

I cannot downgrade rocksdb to 0.12.2 with version 0.8.1.

error: failed to select a version for the requirement `rocksdb = "^0.13"`
  candidate versions found which didn't match: 0.12.2
  location searched: crates.io index
required by package `electrs v0.8.1 (/home/user)`

I guess I'll try again with 0.8.0 or 0.7.1.

dmp1ce · 2019-11-30T00:20:23Z

I have an old 2009 AMD, but I'm not sure if that is my issue with rocksdb. I get Illegal instruction (core dumped) right away when running cargo run --release.

Although, I'm running in docker without a toml configuration.

dmp1ce · 2019-11-30T00:36:32Z

cargo update -p rocksdb --precise 0.12.2 on 0.8.0 didn't help me. I still got Illegal instruction (core dumped).

0.7.1 wouldn't compile for me.

Updating crates.io index
error: failed to write /home/user/Cargo.lock

dmp1ce · 2019-11-30T15:09:35Z

Thank you for reopening, but in fact I did get 0.8.0 to work after running cargo update -p rocksdb --precise 0.12.2! I just needed to get my Dockerfile build working properly. So, it seems I am also hitting the rocksdb issue.

kallerosenbaum · 2019-12-01T12:07:15Z

Thank you for reopening, but in fact I did get 0.8.0 to work after running cargo update -p rocksdb --precise 0.12.2! I just needed to get my Dockerfile build working properly. So, it seems I am also hitting the rocksdb issue.

@dmp1ce Did you manage to downgrade rocksdb on 0.8.1 too? I haven't tried that yet.

dmp1ce · 2019-12-01T14:30:56Z

No, because 0.8.1 requires at least 0.13 of rocksdb. I don't know why.

electrs/Cargo.toml

Line 41 in 1498eed

rocksdb = "0.13"

I just used 0.8.0.

romanz · 2019-12-02T08:16:57Z

Let's downgrade to 0.12.2 until rust-rocksdb/rust-rocksdb#327 is resolved.

#193

romanz · 2019-12-06T11:24:43Z

0.8.2 should fix this issue.

The v0.13 release does not work well with some old CPU models, see: romanz#193 rust-rocksdb/rust-rocksdb#327 rust-rocksdb/rust-rocksdb#363

Kixunil · 2020-08-08T14:51:13Z

Could this be bckported? Blockstream@a03813d

I'm looking at compiling rocksdb with system library and having the dependency pinned makes it harder.

Kixunil · 2020-11-02T12:06:39Z

FYI, I managed to compile electrs with system libraries using this patch: rust-rocksdb/rust-rocksdb#470

If it gets merged, then my issue will be resolved.

romanz · 2021-10-03T08:57:19Z

IIUC, this should currently be resolved in 0.9.0.
Please reopen if it's not :)

romanz self-assigned this Nov 15, 2019

romanz added the bug Something isn't working label Nov 15, 2019

romanz added the build label Nov 25, 2019

romanz closed this as completed Nov 29, 2019

romanz reopened this Nov 30, 2019

romanz added a commit that referenced this issue Dec 2, 2019

Downgrade rocksdb to 0.12.2 to support more x86 CPUs

5833a9a

#193

romanz closed this as completed Dec 6, 2019

shesek added a commit to Blockstream/electrs that referenced this issue Mar 18, 2020

Add "oldcpu" feature to switch to rocksdb v0.12

b4cce60

The v0.13 release does not work well with some old CPU models, see: romanz#193 rust-rocksdb/rust-rocksdb#327 rust-rocksdb/rust-rocksdb#363

shesek added a commit to Blockstream/electrs that referenced this issue Mar 18, 2020

Add "oldcpu" feature to switch to rocksdb v0.12

a03813d

The v0.13 release does not work well with some old CPU models, see: romanz#193 rust-rocksdb/rust-rocksdb#327 rust-rocksdb/rust-rocksdb#363

lukechilds mentioned this issue Apr 11, 2020

electrs exits with code 135 #226

Closed

romanz reopened this Aug 8, 2020

shesek mentioned this issue Jan 15, 2021

std::bad_alloc keep getting this after ~ 30min Blockstream/esplora#142

Open

romanz closed this as completed Oct 3, 2021

romanz mentioned this issue Oct 6, 2021

Bug: Upgraded electrs 0.8.11 --> 0.9.0 and when start the core-dump #536

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Core dump on startup [0.8.0] #193

Core dump on startup [0.8.0] #193

kallerosenbaum commented Nov 14, 2019 •

edited

Loading

kallerosenbaum commented Nov 14, 2019 •

edited

Loading

romanz commented Nov 15, 2019

kallerosenbaum commented Nov 19, 2019

romanz commented Nov 19, 2019 •

edited

Loading

romanz commented Nov 19, 2019

kallerosenbaum commented Nov 19, 2019

kallerosenbaum commented Nov 19, 2019

romanz commented Nov 19, 2019

kallerosenbaum commented Nov 19, 2019

romanz commented Nov 19, 2019

romanz commented Nov 19, 2019 •

edited

Loading

kallerosenbaum commented Nov 20, 2019

romanz commented Nov 20, 2019 •

edited

Loading

kallerosenbaum commented Nov 20, 2019 •

edited

Loading

kallerosenbaum commented Nov 20, 2019

kallerosenbaum commented Nov 20, 2019 •

edited

Loading

kallerosenbaum commented Nov 20, 2019

romanz commented Nov 25, 2019

romanz commented Nov 25, 2019

dmp1ce commented Nov 29, 2019 •

edited

Loading

dmp1ce commented Nov 30, 2019

dmp1ce commented Nov 30, 2019 •

edited

Loading

dmp1ce commented Nov 30, 2019

kallerosenbaum commented Dec 1, 2019

dmp1ce commented Dec 1, 2019

romanz commented Dec 2, 2019

romanz commented Dec 6, 2019

Kixunil commented Aug 8, 2020

Kixunil commented Nov 2, 2020

romanz commented Oct 3, 2021

Core dump on startup [0.8.0] #193

Core dump on startup [0.8.0] #193

Comments

kallerosenbaum commented Nov 14, 2019 • edited Loading

kallerosenbaum commented Nov 14, 2019 • edited Loading

romanz commented Nov 15, 2019

kallerosenbaum commented Nov 19, 2019

romanz commented Nov 19, 2019 • edited Loading

romanz commented Nov 19, 2019

kallerosenbaum commented Nov 19, 2019

kallerosenbaum commented Nov 19, 2019

romanz commented Nov 19, 2019

kallerosenbaum commented Nov 19, 2019

romanz commented Nov 19, 2019

romanz commented Nov 19, 2019 • edited Loading

kallerosenbaum commented Nov 20, 2019

romanz commented Nov 20, 2019 • edited Loading

kallerosenbaum commented Nov 20, 2019 • edited Loading

kallerosenbaum commented Nov 20, 2019

kallerosenbaum commented Nov 20, 2019 • edited Loading

kallerosenbaum commented Nov 20, 2019

romanz commented Nov 25, 2019

romanz commented Nov 25, 2019

dmp1ce commented Nov 29, 2019 • edited Loading

dmp1ce commented Nov 30, 2019

dmp1ce commented Nov 30, 2019 • edited Loading

dmp1ce commented Nov 30, 2019

kallerosenbaum commented Dec 1, 2019

dmp1ce commented Dec 1, 2019

romanz commented Dec 2, 2019

romanz commented Dec 6, 2019

Kixunil commented Aug 8, 2020

Kixunil commented Nov 2, 2020

romanz commented Oct 3, 2021

kallerosenbaum commented Nov 14, 2019 •

edited

Loading

kallerosenbaum commented Nov 14, 2019 •

edited

Loading

romanz commented Nov 19, 2019 •

edited

Loading

romanz commented Nov 19, 2019 •

edited

Loading

romanz commented Nov 20, 2019 •

edited

Loading

kallerosenbaum commented Nov 20, 2019 •

edited

Loading

kallerosenbaum commented Nov 20, 2019 •

edited

Loading

dmp1ce commented Nov 29, 2019 •

edited

Loading

dmp1ce commented Nov 30, 2019 •

edited

Loading