# \[resolved\] Building MUSL C version segmentation fault

**URL:** https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783
**Category:** Third Party Help
**Created:** [July 13, 2023, 8:36pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783 "2023-07-13T20:36:03Z")
**Posts on this page:** 17
**Page:** 1

<div class="post-metadata">

### Author: ![Colin](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@Colin](https://vaultwarden.discourse.group/u/Colin)
#### Post date: [July 13, 2023, 8:36pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/1 "2023-07-13T20:36:03Z")

</div>

Hello everyone,

I’m trying to set up Vaultwarden inside of an Alpine LXC container on Proxmox. I wanted to use Musl Libc as this is what Alpine ships with. I compiled Bitwarden inside of an Alpine LXC container itself to avoid any problems and it works fine.

Just running the executable without any arguments just prints me some help information. But once I add a database URL, it is getting a segmentation fault right after the banner:

```txt
colin-vault:~$ DATABASE_URL=postgresql: USE_SYSLOG=true ./vaultwarden-debug
/--------------------------------------------------------------------\
| Starting Vaultwarden |
| Version 1.29.0-61f90818 |
|--------------------------------------------------------------------|
| This is an *unofficial* Bitwarden implementation, DO NOT use the |
| official channels to report bugs/features, regardless of client. |
| Send usage/configuration questions or feature requests to: |
| https://github.com/dani-garcia/vaultwarden/discussions or |
| https://vaultwarden.discourse.group/ |
| Report suspected bugs/issues in the software itself at: |
| https://github.com/dani-garcia/vaultwarden/issues/new |
\--------------------------------------------------------------------/

Segmentation fault

```

Above is a minimal example. These are all packages I installed manually (Alpine 3.18):  
`ca-certificates curl postgresql15-client openssl`

I don’t know why this is happening and don’t know where to start searching. Could this be due to a missing shared object / package?

PS: I built the binary for debugging and the outcome is the same.

---

<div class="post-metadata">

### Author: ![BlackDex](https://yyz2.discourse-cdn.com/free1/user_avatar/vaultwarden.discourse.group/blackdex/32/8_2.png) [@BlackDex](https://vaultwarden.discourse.group/u/BlackDex)
#### Post date: [July 13, 2023, 8:57pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/2 "2023-07-13T20:57:31Z")

</div>

Try to run with log level trace, and try to set `RUST_BACKTRACE=full` to see of that gives some more info.

It could also be an lxc issue, or a library indeed, but that should produce error messages i think.

---

<div class="post-metadata">

### Author: ![Colin](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@Colin](https://vaultwarden.discourse.group/u/Colin)
#### Post date: [July 13, 2023, 8:59pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/3 "2023-07-13T20:59:14Z")

</div>

Thanks for the tip!

I ran `DATABASE_URL=postgresql: RUST_BACKTRACE=full ./vaultwarden-debug`, but sadly I am getting exactly the same result.

---

<div class="post-metadata">

### Author: ![BlackDex](https://yyz2.discourse-cdn.com/free1/user_avatar/vaultwarden.discourse.group/blackdex/32/8_2.png) [@BlackDex](https://vaultwarden.discourse.group/u/BlackDex)
#### Post date: [July 13, 2023, 9:00pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/4 "2023-07-13T21:00:53Z")

</div>

Could you try to build with sqlite and see what that does?

---

<div class="post-metadata">

### Author: ![Colin](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@Colin](https://vaultwarden.discourse.group/u/Colin)
#### Post date: [July 13, 2023, 9:16pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/5 "2023-07-13T21:16:45Z")

</div>

I am going to do that next. In the meanwhile I used rust-gdb to get a backtrace. I’m still making sense of it but here it is:

> **Backtrace**
>
> ```rs
> Thread 1 "vaultwarden" received signal SIGSEGV, Segmentation fault.
> 0x0000000000000000 in ?? ()
> (gdb) backtrace 
> #0 0x0000000000000000 in ?? ()
> #1 0x00007ffff64ef88f in openssl_sys::openssl::init::{closure#0} () at src/lib.rs:122
> #2 0x00007ffff64eff87 in std::sync::once::{impl#2}::call_once::{closure#0}<openssl_sys::openssl::init::{closure_env#0}> ()
> at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sync/once.rs:149
> #3 0x00007ffff64f049d in std::sys_common::once::futex::Once::call<std::sync::once::{impl#2}::call_once::{closure_env#0}<openssl_sys::openssl::init::{closure_env#0}>> (
> self=0x7ffff7ffc310 <openssl_sys::openssl::INIT::h70a64086eb17aa00>, ignore_poisoning=false, f=0x7ffffffd7718)
> at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/once/futex.rs:124
> #4 0x00007ffff64efecd in std::sync::once::Once::call_once<openssl_sys::openssl::init::{closure_env#0}> (self=0x7ffff7ffc310 <openssl_sys::openssl::INIT::h70a64086eb17aa00>, f=...)
> at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sync/once.rs:149
> #5 0x00007ffff64ef840 in openssl_sys::openssl::init () at src/lib.rs:121
> #6 0x00007ffff64ec794 in openssl::bn::BigNum::new () at src/bn.rs:901
> #7 0x00007ffff64ec85c in openssl::bn::BigNum::from_u32 (n=65537) at src/bn.rs:921
> #8 0x00007ffff64e5619 in openssl::rsa::Rsa<openssl::pkey::Private>::generate (bits=2048) at src/rsa.rs:528
> #9 0x00007ffff552b520 in vaultwarden::check_rsa_keys () at src/main.rs:463
> #10 0x00007ffff51c2957 in vaultwarden::main::{async_block#0} () at src/main.rs:102
> #11 0x00007ffff4b0f273 in tokio::runtime::park::{impl#4}::block_on::{closure#0}<vaultwarden::main::{async_block_env#0}> ()
> at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/park.rs:283
> #12 0x00007ffff4b0dc44 in tokio::runtime::coop::with_budget<core::task::poll::Poll<core::result::Result<(), vaultwarden::error::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<vaultwarden::main::{async_block_env#0}>> (budget=..., f=...) at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/coop.rs:107
> #13 tokio::runtime::coop::budget<core::task::poll::Poll<core::result::Result<(), vaultwarden::error::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<vaultwarden::main::{async_block_env#0}>> (f=...) at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/coop.rs:73
> #14 tokio::runtime::park::CachedParkThread::block_on<vaultwarden::main::{async_block_env#0}> (self=0x7ffffffe2ec8, f=...)
> at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/park.rs:283
> #15 0x00007ffff4a6b420 in tokio::runtime::context::blocking::BlockingRegionGuard::block_on<vaultwarden::main::{async_block_env#0}> (self=0x7ffffffe8fa0, f=...)
> at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context/blocking.rs:66
> #16 0x00007ffff4516f62 in tokio::runtime::scheduler::multi_thread::{impl#0}::block_on::{closure#0}<vaultwarden::main::{async_block_env#0}> (blocking=0x7ffffffe8fa0)
> at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/multi_thread/mod.rs:87
> #17 0x00007ffff4cdc117 in tokio::runtime::context::runtime::enter_runtime<tokio::runtime::scheduler::multi_thread::{impl#0}::block_on::{closure_env#0}<vaultwarden::main::{async_block_env#0}>, core::result::Result<(), vaultwarden::error::Error>> (handle=0x7fffffff4ff0, allow_block_in_place=true, f=...)
> at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context/runtime.rs:65
> #18 0x00007ffff4516ede in tokio::runtime::scheduler::multi_thread::MultiThread::block_on<vaultwarden::main::{async_block_env#0}> (self=0x7fffffff4fc8, handle=0x7fffffff4ff0, future=...)
> at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/multi_thread/mod.rs:86
> #19 0x00007ffff517d8c3 in tokio::runtime::runtime::Runtime::block_on<vaultwarden::main::{async_block_env#0}> (self=0x7fffffff4fb0, future=...)
> at /home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/runtime.rs:313
> #20 0x00007ffff537489c in rocket::async_run<vaultwarden::main::{async_block_env#0}, core::result::Result<(), vaultwarden::error::Error>> (fut=..., workers=8, sync=512, force_end=true, 
> name="rocket-worker-thread") at /home/builder/.cargo/git/checkouts/rocket-8bf16d9ca7e90bdc/ce441b5/core/lib/src/lib.rs:247
> #21 0x00007ffff5374203 in rocket::async_main<core::result::Result<(), vaultwarden::error::Error>, vaultwarden::main::{async_block_env#0}> (fut=...)
> at /home/builder/.cargo/git/checkouts/rocket-8bf16d9ca7e90bdc/ce441b5/core/lib/src/lib.rs:279
> #22 0x00007ffff552c87c in vaultwarden::main () at src/main.rs:91
> #23 0x00007ffff453b142 in core::ops::function::FnOnce::call_once<fn() -> core::result::Result<(), vaultwarden::error::Error>, ()> ()
> 
> ```

---

<div class="post-metadata">

### Author: ![BlackDex](https://yyz2.discourse-cdn.com/free1/user_avatar/vaultwarden.discourse.group/blackdex/32/8_2.png) [@BlackDex](https://vaultwarden.discourse.group/u/BlackDex)
#### Post date: [July 13, 2023, 9:28pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/6 "2023-07-13T21:28:37Z")

</div>

Looks like it crashes during the creation of the rsa pem generation.  
Either openssl has some issues in some way. Or something else breaks.

You could try to use the `vendored_openssl` feature and see if that helps.

---

<div class="post-metadata">

### Author: ![Colin](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@Colin](https://vaultwarden.discourse.group/u/Colin)
#### Post date: [July 13, 2023, 9:32pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/7 "2023-07-13T21:32:59Z")

</div>

This is the code that causes the null pointer call:

```rs
    pub fn init() {
        use std::ptr;

        #[cfg(not(ossl111b))]
        let init_options = OPENSSL_INIT_LOAD_SSL_STRINGS;
        #[cfg(ossl111b)]
        let init_options = OPENSSL_INIT_LOAD_SSL_STRINGS | OPENSSL_INIT_NO_ATEXIT;

        INIT.call_once(|| unsafe {
            OPENSSL_init_ssl(init_options, ptr::null_mut());
        })
    }

```

I attempted to rebuild the binary with the `vendored_openssl` flag, and the build script fails because it can’t find openssl. It seems that it either can’t find my openssl install or I am missing a package.

---

<div class="post-metadata">

### Author: ![BlackDex](https://yyz2.discourse-cdn.com/free1/user_avatar/vaultwarden.discourse.group/blackdex/32/8_2.png) [@BlackDex](https://vaultwarden.discourse.group/u/BlackDex)
#### Post date: [July 13, 2023, 9:54pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/8 "2023-07-13T21:54:08Z")

</div>

Strange, as vendored should build it it self.  
Do you have libssl3 installed?

---

<div class="post-metadata">

### Author: ![Colin](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@Colin](https://vaultwarden.discourse.group/u/Colin)
#### Post date: [July 13, 2023, 9:59pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/9 "2023-07-13T21:59:50Z")

</div>

I found the issue. I did install gcc etc, but make was missing. The error message it gave me was a bit misleading. Vaultwarden starts without any issues when run with vendored openssl, I would still like to figure out why it cannot link to it though.

---

<div class="post-metadata">

### Author: ![Colin](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@Colin](https://vaultwarden.discourse.group/u/Colin)
#### Post date: [July 13, 2023, 10:16pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/10 "2023-07-13T22:16:01Z")

</div>

I was able to create a minimal project to reproduce this issue without bitwarden:

```rs
use openssl_sys::init;

fn main() {
    println!("calling init");
    init();
    println!("done");
}

```

I will create an issue at `openssl-sys` as this is not an issue with Vaultwarden.

Should I close this topic @BlackDex?

---

<div class="post-metadata">

### Author: ![BlackDex](https://yyz2.discourse-cdn.com/free1/user_avatar/vaultwarden.discourse.group/blackdex/32/8_2.png) [@BlackDex](https://vaultwarden.discourse.group/u/BlackDex)
#### Post date: [July 13, 2023, 10:19pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/11 "2023-07-13T22:19:39Z")

</div>

Strange that it works with vendored and not native

Which version of Alpine are you using?

---

<div class="post-metadata">

### Author: ![Colin](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@Colin](https://vaultwarden.discourse.group/u/Colin)
#### Post date: [July 13, 2023, 10:20pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/12 "2023-07-13T22:20:26Z")

</div>

I found this older issue which found out that apparently ~~Alpine~~ Rust is to blame for a broken openssl build:

> <https://github.com/est31/cargo-udeps/issues/89>
>
> My environment:
> \`\`\`
> + rustup --version
> rustup 1.23.1 (2020-11-30)
> ...
> cargo… 1.50.0 (f04e7fab7 2021-02-04)
> \`\`\`
> The output from the command when running in a workspace with ~15 crates:
> \`\`\`
> $ rustup show
> Default host: x86\_64-unknown-linux-musl
> rustup home: /usr/local/rustup
> installed toolchains
> \--------------------
> nightly-x86\_64-unknown-linux-musl
> 1.50.0-x86\_64-unknown-linux-musl (default)
> active toolchain
> \----------------
> 1.50.0-x86\_64-unknown-linux-musl (default)
> rustc 1.50.0 (cb75ad5db 2021-02-10)
> 
> $ cargo +nightly udeps --release --all-targets
> /bin/bash: line 147: 26 Segmentation fault (core dumped) cargo +nightly udeps --release --all-targets
> \`\`\`
> Any advice on how to resolve this issue? Thanks.

> The system-installed copy of OpenSSL dynamically links to the system MUSL, but Rust will by default statically link to a bundled MUSL, and that causes things to break when calling into OpenSSL.

> [@BlackDex](#):
>
> Which version of Alpine are you using?

Alpine 3.18

---

<div class="post-metadata">

### Author: ![BlackDex](https://yyz2.discourse-cdn.com/free1/user_avatar/vaultwarden.discourse.group/blackdex/32/8_2.png) [@BlackDex](https://vaultwarden.discourse.group/u/BlackDex)
#### Post date: [July 13, 2023, 10:34pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/13 "2023-07-13T22:34:36Z")

</div>

Well, that is a very old rust version, doubt it still is an issue

---

<div class="post-metadata">

### Author: ![Colin](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@Colin](https://vaultwarden.discourse.group/u/Colin)
#### Post date: [July 13, 2023, 10:36pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/14 "2023-07-13T22:36:14Z")

</div>

Apparently they changed it to link dynamically in the meantime:  
`https://github.com/rust-lang/compiler-team/issues/422` (“An error occurred: Sorry you cannot post a link to that host.”)

---

<div class="post-metadata">

### Author: ![Colin](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@Colin](https://vaultwarden.discourse.group/u/Colin)
#### Post date: [July 13, 2023, 11:52pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/15 "2023-07-13T23:52:48Z")

</div>

Progress update on locating the issue: It’s a linking problem of the Rust compiler on musl:  
`https://github.com/rust-lang/rust/issues/82193`

I now got the same issue with postgresql, as it calls null instead of calling the linked library function. Also I don’t think this list should be empty:

```txt
(gdb) info sharedlibrary
No shared libraries loaded at this time.

```

Good thing is that the “workaround” mentioned in the issue does work for me:

```txt
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0x00007ffff7f78070 0x00007ffff7fc3357 Yes (*) /lib/ld-musl-x86_64.so.1
0x00007ffff7f17ff0 0x00007ffff7f37b6f Yes (*) /usr/lib/libpq.so.5
0x00007ffff7ef5330 0x00007ffff7f0a835 Yes (*) /usr/lib/libgcc_s.so.1
0x00007ffff7e7eeb0 0x00007ffff7ec6b30 Yes (*) /lib/libssl.so.3
0x00007ffff7aa3000 0x00007ffff7d3e192 Yes (*) /lib/libcrypto.so.3

```

I created the file `.cargo/config.toml` with the following content:

```toml
[target."x86_64-unknown-linux-musl"]
rustflags = ["-C", "target-feature=-crt-static"]

```

I can now compile using `cargo build --features postgresql`.

---

<div class="post-metadata">

### Author: ![Colin](https://avatars.discourse-cdn.com/v4/letter/c/e19b73/32.png) [@Colin](https://vaultwarden.discourse.group/u/Colin)
#### Post date: [July 14, 2023, 1:55pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/16 "2023-07-14T13:55:01Z")

</div>

I asked on the Rust Discord and was told that this is a known compiler bug. Alpine has their own patched version of the Rust toolchain available as packages. When I compiled with the patched compiler, it worked without any issues.

---

<div class="post-metadata">

### Author: ![BlackDex](https://yyz2.discourse-cdn.com/free1/user_avatar/vaultwarden.discourse.group/blackdex/32/8_2.png) [@BlackDex](https://vaultwarden.discourse.group/u/BlackDex)
#### Post date: [July 14, 2023, 2:02pm UTC](https://vaultwarden.discourse.group/t/resolved-building-musl-c-version-segmentation-fault/2783/17 "2023-07-14T14:02:22Z")

</div>

Ah nice.

I actually never build using Alpine it self with the most recent versions.  
I build my own gcc musl build tools and use those to build a MUSL version of Vaultwarden to be used with Alpine base images.

But, i statically build all the libraries and after that Vaultwarden.

Also, a small performance note, you might want to enable the `enable_mimalloc` feature which uses a different Memory Allocator then the default MUSL, which has some known performance issues.
