[resolved] Building MUSL C version segmentation fault

Third Party Help
1

Hello everyone,

I’m trying to set up Vaultwarden inside of an Alpine LXC container on Proxmox. I wanted to use Musl Libc as this is what Alpine ships with. I compiled Bitwarden inside of an Alpine LXC container itself to avoid any problems and it works fine.

Just running the executable without any arguments just prints me some help information. But once I add a database URL, it is getting a segmentation fault right after the banner:

colin-vault:~$ DATABASE_URL=postgresql: USE_SYSLOG=true ./vaultwarden-debug
/--------------------------------------------------------------------\
|                        Starting Vaultwarden                        |
|                      Version 1.29.0-61f90818                       |
|--------------------------------------------------------------------|
| This is an *unofficial* Bitwarden implementation, DO NOT use the   |
| official channels to report bugs/features, regardless of client.   |
| Send usage/configuration questions or feature requests to:         |
|   https://github.com/dani-garcia/vaultwarden/discussions or        |
|   https://vaultwarden.discourse.group/                             |
| Report suspected bugs/issues in the software itself at:            |
|   https://github.com/dani-garcia/vaultwarden/issues/new            |
\--------------------------------------------------------------------/

Segmentation fault

Above is a minimal example. These are all packages I installed manually (Alpine 3.18):
ca-certificates curl postgresql15-client openssl

I don’t know why this is happening and don’t know where to start searching. Could this be due to a missing shared object / package?

PS: I built the binary for debugging and the outcome is the same.

2

Try to run with log level trace, and try to set RUST_BACKTRACE=full to see of that gives some more info.

It could also be an lxc issue, or a library indeed, but that should produce error messages i think.

3

Thanks for the tip!

I ran DATABASE_URL=postgresql: RUST_BACKTRACE=full ./vaultwarden-debug, but sadly I am getting exactly the same result.

4

Could you try to build with sqlite and see what that does?

5

I am going to do that next. In the meanwhile I used rust-gdb to get a backtrace. I’m still making sense of it but here it is:

Backtrace 
Thread 1 "vaultwarden" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) backtrace 
#0  0x0000000000000000 in ?? ()
#1  0x00007ffff64ef88f in openssl_sys::openssl::init::{closure#0} () at src/lib.rs:122
#2  0x00007ffff64eff87 in std::sync::once::{impl#2}::call_once::{closure#0}<openssl_sys::openssl::init::{closure_env#0}> ()
    at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sync/once.rs:149
#3  0x00007ffff64f049d in std::sys_common::once::futex::Once::call<std::sync::once::{impl#2}::call_once::{closure_env#0}<openssl_sys::openssl::init::{closure_env#0}>> (
    self=0x7ffff7ffc310 <openssl_sys::openssl::INIT::h70a64086eb17aa00>, ignore_poisoning=false, f=0x7ffffffd7718)
    at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/once/futex.rs:124
#4  0x00007ffff64efecd in std::sync::once::Once::call_once<openssl_sys::openssl::init::{closure_env#0}> (self=0x7ffff7ffc310 <openssl_sys::openssl::INIT::h70a64086eb17aa00>, f=...)
    at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sync/once.rs:149
#5  0x00007ffff64ef840 in openssl_sys::openssl::init () at src/lib.rs:121
#6  0x00007ffff64ec794 in openssl::bn::BigNum::new () at src/bn.rs:901
#7  0x00007ffff64ec85c in openssl::bn::BigNum::from_u32 (n=65537) at src/bn.rs:921
#8  0x00007ffff64e5619 in openssl::rsa::Rsa<openssl::pkey::Private>::generate (bits=2048) at src/rsa.rs:528
#9  0x00007ffff552b520 in vaultwarden::check_rsa_keys () at src/main.rs:463
#10 0x00007ffff51c2957 in vaultwarden::main::{async_block#0} () at src/main.rs:102
#11 0x00007ffff4b0f273 in tokio::runtime::park::{impl#4}::block_on::{closure#0}<vaultwarden::main::{async_block_env#0}> ()
    at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/park.rs:283
#12 0x00007ffff4b0dc44 in tokio::runtime::coop::with_budget<core::task::poll::Poll<core::result::Result<(), vaultwarden::error::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<vaultwarden::main::{async_block_env#0}>> (budget=..., f=...) at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/coop.rs:107
#13 tokio::runtime::coop::budget<core::task::poll::Poll<core::result::Result<(), vaultwarden::error::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<vaultwarden::main::{async_block_env#0}>> (f=...) at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/coop.rs:73
#14 tokio::runtime::park::CachedParkThread::block_on<vaultwarden::main::{async_block_env#0}> (self=0x7ffffffe2ec8, f=...)
    at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/park.rs:283
#15 0x00007ffff4a6b420 in tokio::runtime::context::blocking::BlockingRegionGuard::block_on<vaultwarden::main::{async_block_env#0}> (self=0x7ffffffe8fa0, f=...)
    at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context/blocking.rs:66
#16 0x00007ffff4516f62 in tokio::runtime::scheduler::multi_thread::{impl#0}::block_on::{closure#0}<vaultwarden::main::{async_block_env#0}> (blocking=0x7ffffffe8fa0)
    at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/multi_thread/mod.rs:87
#17 0x00007ffff4cdc117 in tokio::runtime::context::runtime::enter_runtime<tokio::runtime::scheduler::multi_thread::{impl#0}::block_on::{closure_env#0}<vaultwarden::main::{async_block_env#0}>, core::result::Result<(), vaultwarden::error::Error>> (handle=0x7fffffff4ff0, allow_block_in_place=true, f=...)
    at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context/runtime.rs:65
#18 0x00007ffff4516ede in tokio::runtime::scheduler::multi_thread::MultiThread::block_on<vaultwarden::main::{async_block_env#0}> (self=0x7fffffff4fc8, handle=0x7fffffff4ff0, future=...)
    at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/multi_thread/mod.rs:86
#19 0x00007ffff517d8c3 in tokio::runtime::runtime::Runtime::block_on<vaultwarden::main::{async_block_env#0}> (self=0x7fffffff4fb0, future=...)
    at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/runtime.rs:313
#20 0x00007ffff537489c in rocket::async_run<vaultwarden::main::{async_block_env#0}, core::result::Result<(), vaultwarden::error::Error>> (fut=..., workers=8, sync=512, force_end=true, 
    name="rocket-worker-thread") at /home/builder/.cargo/git/checkouts/rocket-8bf16d9ca7e90bdc/ce441b5/core/lib/src/lib.rs:247
#21 0x00007ffff5374203 in rocket::async_main<core::result::Result<(), vaultwarden::error::Error>, vaultwarden::main::{async_block_env#0}> (fut=...)
    at /home/builder/.cargo/git/checkouts/rocket-8bf16d9ca7e90bdc/ce441b5/core/lib/src/lib.rs:279
#22 0x00007ffff552c87c in vaultwarden::main () at src/main.rs:91
#23 0x00007ffff453b142 in core::ops::function::FnOnce::call_once<fn() -> core::result::Result<(), vaultwarden::error::Error>, ()> ()
6

Looks like it crashes during the creation of the rsa pem generation.
Either openssl has some issues in some way. Or something else breaks.

You could try to use the vendored_openssl feature and see if that helps.

7

This is the code that causes the null pointer call:

    pub fn init() {
        use std::ptr;

        #[cfg(not(ossl111b))]
        let init_options = OPENSSL_INIT_LOAD_SSL_STRINGS;
        #[cfg(ossl111b)]
        let init_options = OPENSSL_INIT_LOAD_SSL_STRINGS | OPENSSL_INIT_NO_ATEXIT;

        INIT.call_once(|| unsafe {
            OPENSSL_init_ssl(init_options, ptr::null_mut());
        })
    }

I attempted to rebuild the binary with the vendored_openssl flag, and the build script fails because it can’t find openssl. It seems that it either can’t find my openssl install or I am missing a package.

8

Strange, as vendored should build it it self.
Do you have libssl3 installed?

9

I found the issue. I did install gcc etc, but make was missing. The error message it gave me was a bit misleading. Vaultwarden starts without any issues when run with vendored openssl, I would still like to figure out why it cannot link to it though.

10

I was able to create a minimal project to reproduce this issue without bitwarden:

use openssl_sys::init;

fn main() {
    println!("calling init");
    init();
    println!("done");
}

I will create an issue at openssl-sys as this is not an issue with Vaultwarden.

Should I close this topic @BlackDex?

11

Strange that it works with vendored and not native

Which version of Alpine are you using?

12

I found this older issue which found out that apparently AlpineRust is to blame for a broken openssl build:

The system-installed copy of OpenSSL dynamically links to the system MUSL, but Rust will by default statically link to a bundled MUSL, and that causes things to break when calling into OpenSSL.

Alpine 3.18

13

Well, that is a very old rust version, doubt it still is an issue

14

Apparently they changed it to link dynamically in the meantime:
https://github.com/rust-lang/compiler-team/issues/422 (“An error occurred: Sorry you cannot post a link to that host.”)

15

Progress update on locating the issue: It’s a linking problem of the Rust compiler on musl:
https://github.com/rust-lang/rust/issues/82193

I now got the same issue with postgresql, as it calls null instead of calling the linked library function. Also I don’t think this list should be empty:

(gdb) info sharedlibrary
No shared libraries loaded at this time.

Good thing is that the “workaround” mentioned in the issue does work for me:

(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x00007ffff7f78070  0x00007ffff7fc3357  Yes (*)     /lib/ld-musl-x86_64.so.1
0x00007ffff7f17ff0  0x00007ffff7f37b6f  Yes (*)     /usr/lib/libpq.so.5
0x00007ffff7ef5330  0x00007ffff7f0a835  Yes (*)     /usr/lib/libgcc_s.so.1
0x00007ffff7e7eeb0  0x00007ffff7ec6b30  Yes (*)     /lib/libssl.so.3
0x00007ffff7aa3000  0x00007ffff7d3e192  Yes (*)     /lib/libcrypto.so.3

I created the file .cargo/config.toml with the following content:

[target."x86_64-unknown-linux-musl"]
rustflags = ["-C", "target-feature=-crt-static"]

I can now compile using cargo build --features postgresql.

16

I asked on the Rust Discord and was told that this is a known compiler bug. Alpine has their own patched version of the Rust toolchain available as packages. When I compiled with the patched compiler, it worked without any issues.

17

Ah nice.

I actually never build using Alpine it self with the most recent versions.
I build my own gcc musl build tools and use those to build a MUSL version of Vaultwarden to be used with Alpine base images.

But, i statically build all the libraries and after that Vaultwarden.

Also, a small performance note, you might want to enable the enable_mimalloc feature which uses a different Memory Allocator then the default MUSL, which has some known performance issues.

1 Like