WebSocket connections fail

Hi there,
I recently noticed an issue where WS connections fail to my Vaultwarden instance. I’m running version 1.27.0 using docker-compose behind an nginx reverse proxy.

docker-compose.yaml

version: '2'

services:
  vaultwarden:
    container_name: vaultwarden
    image: vaultwarden/server:latest
    restart: unless-stopped
    network_mode: bridge
    environment:
      - SIGNUPS_ALLOWED=false
      - ADMIN_TOKEN=<admin_token>
      - DOMAIN=https://<some-domain>/
      - SMTP_HOST=<smtp-host>
      - SMTP_PORT=25
      - SMTP_SSL=false
      - SMTP_FROM=keeper@<some-domain>
      - SMTP_FROM_NAME=Vaultwarden Keeper
      - WEBSOCKET_ENABLED=true
      - SHOW_PASSWORD_HINT=false
    ports:
      - 127.0.0.1:9085:80
      - 127.0.0.1:9086:3012
    volumes:
      - /opt/containers/vaultwarden/data:/data

I copied the relevant location entries for the nginx include directly from the proxy examples “Nginx (by blackdex)” so I don’t copy it over here.

Nginx logs an error when I try to access the /notifications/hub endpoint (some for IPv4 clients and IPv6 clients):

Feb 06 09:30:55 <host>.<some-domain> nginx[3879602]: 2023/02/06 09:30:55 [error] 3879602#3879602: *11 upstream prematurely closed connection while reading response header from upstream, client: <my-ip>, server: <some-domain>, request: "GET /notifications/hub?access_token=<lengthy-token> HTTP/1.1", upstream: "http://127.0.0.1:9086/notifications/hub?access_token=<lengthy-token>

At the same time the docker logs contain this line:

[2023-02-06 09:30:55.764][vaultwarden::api::notifications][INFO] Accepting WS connection from 172.17.0.1:51454

Yes, according to the logs the WS server is starting, no errors:

[2023-02-06 09:03:48.372][vaultwarden::api::notifications][INFO] Starting WebSockets server on 0.0.0.0:3012
[2023-02-06 09:03:48.376][start][INFO] Rocket has launched from http://0.0.0.0:80

I also found references to a curl command that should yield an error 400 and/or an empty messages struct, however that only returns nothing (empty reply from server):

$ docker exec -it vaultwarden curl 127.0.0.1:3012 -kvvv -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Host: <some-domain>"  
*   Trying 127.0.0.1:3012...
* Connected to 127.0.0.1 (127.0.0.1) port 3012 (#0)
> GET / HTTP/1.1
> Host: <some-domain>
> User-Agent: curl/7.74.0
> Accept: */*
> Connection: Upgrade
> Upgrade: websocket
> 
* Empty reply from server
* Connection #0 to host 127.0.0.1 left intact
curl: (52) Empty reply from server

Is anybody observing similar issues, or am I just missing something here?
I think I remember it was working with one of the older versions, when I tested the setup initially but I don’t recall the exact version.

Thanks in advance,
Sebastian

I had similar issue in the past. As my self-hosted services keeps growing, I decided long time ago to migrate from Apache2 or Nginx to HAProxy. Apache and Nginx are good for webserver, but for reverse proxy it is HAProxy which gave me the biggest capabilities, speed and ease of configuration. Here is my config from HAProxy:

frontend https-in
    bind *:80 name http
    bind *:443 ssl crt-list /etc/haproxy/crt-list.cfg alpn h2,http/1.1 name https
    mode http

    use_backend bitwarden_ws_backend      if { hdr(Host) -i bitwarden.domain.tld } !{ path_beg /notif>
    use_backend bitwarden_backend         if { hdr(Host) -i bitwarden.domain.tld}


backend bitwarden_ws_backend
    server bitwarden_ws_server 192.168.80.112:3012
    http-request redirect scheme https code 301 if !{ ssl_fc }

backend bitwarden_backend
    server bitwarden_server 192.168.80.112:449 check
    http-request redirect scheme https code 301 if !{ ssl_fc }
    http-request deny if { path_beg /admin }

Thanks for the suggestion, I’ll give it a try.

For consideration: I honestly do not think there is an issue with the reverse proxy configuration. I get the same “empty reply” from the reverse proxy that I also got from the websocket endpoint inside the container. Is it expected or is your output on docker exec -it vaultwarden curl 127.0.0.1:3012 -kvvv -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Host: <some-domain>" a different one (like bad request)?

Here is an output:

user@host:~$ docker exec -it vaultwarden curl 127.0.0.1:3012 -kvvv -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Host: bitwarden.mydomain.tld"
*   Trying 127.0.0.1:3012...
* Connected to 127.0.0.1 (127.0.0.1) port 3012 (#0)
> GET / HTTP/1.1
> Host: bitwarden.mydomain.tld
> User-Agent: curl/7.74.0
> Accept: */*
> Connection: Upgrade
> Upgrade: websocket
> 
* Empty reply from server
* Connection #0 to host 127.0.0.1 left intact
curl: (52) Empty reply from server

and check this:

user@host:~$ curl https://bitwarden.mydomain.tld -kvvv -H "Connection: Upgrade" -H "Upgrade: websocket" 
*   Trying 123.456.789.012:443...
* Connected to bitwarden.mydomain.tld (123.456.789.012) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=bitwarden.mydomain.tld
*  start date: Jan 19 03:22:49 2023 GMT
*  expire date: Apr 19 03:22:48 2023 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x558e636da960)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: bitwarden.mydomain.tld
> user-agent: curl/7.81.0
> accept: */*
> connection: Upgrade
> upgrade: websocket
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200 
< content-type: text/html; charset=utf-8
< cache-control: public, max-age=600
< expires: Sat, 18 Feb 2023 16:17:45 GMT
< server: Rocket
< x-xss-protection: 0
< content-security-policy: default-src 'self'; base-uri 'self'; form-action 'self'; object-src 'self' blob:; script-src 'self'; style-src 'self' 'unsafe-inline'; child-src 'self' https://*.duosecurity.com https://*.duofederal.com; frame-src 'self' https://*.duosecurity.com https://*.duofederal.com; frame-ancestors 'self' chrome-extension://nngceckbapebfimnlniiiahkandclblb chrome-extension://jbkfoedolllekgbhcbcoahefnbanhhlh moz-extension://* ; img-src 'self' data: https://haveibeenpwned.com https://www.gravatar.com ; connect-src 'self' https://api.pwnedpasswords.com https://2fa.directory https://app.simplelogin.io/api/ https://app.anonaddy.com/api/ https://api.fastmail.com/ ;
< content-length: 1240
< date: Sat, 18 Feb 2023 16:07:45 GMT
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< strict-transport-security: max-age=31536000;includeSubDomains;preload
< referrer-policy: same-origin
< permissions-policy: accelerometer=(),autoplay=(self),camera=(self),encrypted-media=(self),fullscreen=*,geolocation=(self),gyroscope=(),magnetometer=(),microphone=(self),midi=(),payment=(),sync-xhr=*,usb=(self),xr-spatial-tracking=()
< 
<!doctype html><html class="theme_light"><head><meta charset="utf-8"/><meta name="viewport" content="width=1010"/><meta name="theme-color" content="#175DDC"/><title page-title>Vaultwarden Web Vault</title><link rel="apple-touch-icon" sizes="180x180" href="images/apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="images/favicon-32x32.png"/><link rel="icon" type="image/png" sizes="16x16" href="images/favicon-16x16.png"/><link rel="mask-icon" href="images/safari-pinned-tab.svg" color="#175DDC"/><link rel="manifest" href="ca8f66ed7fccfcd0809f.json"/><script defer="defer" src="theme_head.5f24ba8d7aa944e6f52b.js"></script><link href="app/main.82096a4e78d5d3f7b01b.css" rel="stylesheet"></head><body class="layout_frontend"><app-root><div class="mt-5 d-flex justify-content-center"><div><img class="mb-4 logo logo-themed" alt="Bitwarden"/><p class="text-center"><i class="bwi bwi-spinner bwi-spin bwi-2x text-muted" title="Loading" aria-hidden="true"></i></p></div></div></app-root><script defer="* Connection #0 to host bitwarden.mydomain.tld left intact
defer" src="app/polyfills.428c25638840333a09ee.js"></script><script defer="defer" src="app/vendor.7c30c6e2b5ba56506ea9.js"></script><script defer="defer" src="app/main.5f8690f5c03a207c390a.js"></script></body></html>

Didn’t have much time to look into this (apparently rather “my”) issue … tested a bit with HAProxy back and forth on a different port but left my nginx config untouched.
Well, tried again today and voila, the issue disappeared and I didn’t have to change a single bit, so in the end I can blame it on client misbehaviour and general cosmic radiation.

For my own records: yes, empty reply from server seems to be expected and acceptable given the input from the curl request.

Thanks for all who took their time to look into this issue!