Websocket notifications not working with Caddy 2

Help
1

I can’t seem to get websocket notifications to work with bitwarden_rs behind my Caddy 2 reverse proxy.
Caddy log is constantly spammed with this error:

Oct 01 14:23:46 my-server caddy[26985]: {"level":"error","ts":1601551426.1136703,"logger":"http.log.error","msg":"read tcp 127.0.0.1:49296->127.0.0.1:3012: read: connection reset by peer","request":{"remote_addr":"79.177.86.25:60170","proto":"HTTP/1.1","method":"GET","host":"bitwarden.my.server","uri":"/notifications/hub?access_token=%VERYLONGTOKEN%","headers":{"Connection":["Upgrade"],"Accept-Language":["en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"],"Sec-Websocket-Key":["z0Gkdt8VLpCisCyRX9BQ4w=="],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Accept-Encoding":["gzip, deflate, br"],"Pragma":["no-cache"],"Cache-Control":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"],"Upgrade":["websocket"],"Origin":["chrome-extension://nngceckbapebfimnlniiiahkandclblb"],"Sec-Websocket-Version":["13"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"bitwarden.my.server"}},"duration":0.000925743,"status":502,"err_id":"28mat83yq","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:440)"}
Oct 01 14:18:53 my-server caddy[26985]: {"level":"error","ts":1601551133.9927924,"logger":"http.log.error","msg":"read tcp 127.0.0.1:49028->127.0.0.1:3012: read: connection reset by peer","request":{"remote_addr":"79.177.86.25:60147","proto":"HTTP/1.1","method":"GET","host":"bitwarden.my.server","uri":"/notifications/hub?access_token=%VERYLONGTOKEN%","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"],"Origin":["chrome-extension://nngceckbapebfimnlniiiahkandclblb"],"Sec-Websocket-Version":["13"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"],"Sec-Websocket-Key":["NAwMuCHAtDsvZ1ZKy+v4XA=="],"Pragma":["no-cache"],"Connection":["Upgrade"],"Cache-Control":["no-cache"],"Upgrade":["websocket"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"bitwarden.my.server"}},"duration":0.000797282,"status":502,"err_id":"ueavyvwjn","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:440)"}
Oct 01 14:26:58 my-server caddy[26985]: {"level":"error","ts":1601551618.4289227,"logger":"http.log.error","msg":"read tcp 127.0.0.1:49582->127.0.0.1:3012: read: connection reset by peer","request":{"remote_addr":"79.177.86.25:60188","proto":"HTTP/1.1","method":"GET","host":"bitwarden.my.server","uri":"/notifications/hub?access_token=%VERYLONGTOKEN%","headers":{"Pragma":["no-cache"],"Cache-Control":["no-cache"],"Accept-Language":["en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"],"Sec-Websocket-Key":["igUKGtz5AhPkkU+ptC/Lhg=="],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Connection":["Upgrade"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"],"Upgrade":["websocket"],"Origin":["chrome-extension://nngceckbapebfimnlniiiahkandclblb"],"Sec-Websocket-Version":["13"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"bitwarden.my.server"}},"duration":0.000845008,"status":502,"err_id":"8yj8jgj00","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:440)"}
Oct 01 14:29:20 my-server caddy[26985]: {"level":"error","ts":1601551760.9370563,"logger":"http.log.error","msg":"read tcp 127.0.0.1:49804->127.0.0.1:3012: read: connection reset by peer","request":{"remote_addr":"79.177.86.25:60207","proto":"HTTP/1.1","method":"GET","host":"bitwarden.my.server","uri":"/notifications/hub?access_token=%VERYLONGTOKEN%","headers":{"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Pragma":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"],"Upgrade":["websocket"],"Sec-Websocket-Version":["13"],"Accept-Language":["en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"],"Sec-Websocket-Key":["OGeDCzJeGMFWdO8ItOVGew=="],"Connection":["Upgrade"],"Cache-Control":["no-cache"],"Origin":["chrome-extension://nngceckbapebfimnlniiiahkandclblb"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"bitwarden.my.server"}},"duration":0.054073999,"status":502,"err_id":"ebydfuzs0","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:440)"}
Oct 01 14:32:27 my-server caddy[26985]: {"level":"error","ts":1601551947.5327892,"logger":"http.log.error","msg":"read tcp 127.0.0.1:49872->127.0.0.1:3012: read: connection reset by peer","request":{"remote_addr":"79.177.86.25:60236","proto":"HTTP/1.1","method":"GET","host":"bitwarden.my.server","uri":"/notifications/hub?access_token=%VERYLONGTOKEN%","headers":{"Accept-Language":["en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"],"Connection":["Upgrade"],"Pragma":["no-cache"],"Cache-Control":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"],"Origin":["chrome-extension://nngceckbapebfimnlniiiahkandclblb"],"Sec-Websocket-Version":["13"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Websocket-Key":["O9npeqSlb8k3ZlKnpkFycg=="],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Upgrade":["websocket"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"bitwarden.my.server"}},"duration":0.025380758,"status":502,"err_id":"qwu7wcspp","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:440)"}
Oct 01 14:34:28 my-server caddy[26985]: {"level":"error","ts":1601552068.7511015,"logger":"http.log.error","msg":"read tcp 127.0.0.1:50092->127.0.0.1:3012: read: connection reset by peer","request":{"remote_addr":"79.177.86.25:60239","proto":"HTTP/1.1","method":"GET","host":"bitwarden.my.server","uri":"/notifications/hub?access_token=%VERYLONGTOKEN%","headers":{"Origin":["chrome-extension://nngceckbapebfimnlniiiahkandclblb"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Websocket-Key":["NdeN6rH3/hINGSIL7DF57w=="],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Cache-Control":["no-cache"],"Upgrade":["websocket"],"Sec-Websocket-Version":["13"],"Accept-Language":["en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"],"Connection":["Upgrade"],"Pragma":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"bitwarden.my.server"}},"duration":0.000908832,"status":502,"err_id":"rq5c16wee","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:440)"}
Oct 01 14:36:46 my-server caddy[26985]: {"level":"error","ts":1601552206.3491156,"logger":"http.log.error","msg":"read tcp 127.0.0.1:50136->127.0.0.1:3012: read: connection reset by peer","request":{"remote_addr":"79.177.86.25:60248","proto":"HTTP/1.1","method":"GET","host":"bitwarden.my.server","uri":"/notifications/hub?access_token=%VERYLONGTOKEN%","headers":{"Cache-Control":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"],"Upgrade":["websocket"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Sec-Websocket-Key":["kF3VwPdwgUsiMcPPWPf/ew=="],"Connection":["Upgrade"],"Pragma":["no-cache"],"Origin":["chrome-extension://nngceckbapebfimnlniiiahkandclblb"],"Sec-Websocket-Version":["13"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"bitwarden.my.server"}},"duration":0.000921381,"status":502,"err_id":"5rq2kw7d6","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:440)"}
Oct 01 14:39:58 my-server caddy[26985]: {"level":"error","ts":1601552398.0302348,"logger":"http.log.error","msg":"read tcp 127.0.0.1:50372->127.0.0.1:3012: read: connection reset by peer","request":{"remote_addr":"79.177.86.25:60252","proto":"HTTP/1.1","method":"GET","host":"bitwarden.my.server","uri":"/notifications/hub?access_token=%VERYLONGTOKEN%","headers":{"Connection":["Upgrade"],"Origin":["chrome-extension://nngceckbapebfimnlniiiahkandclblb"],"Sec-Websocket-Version":["13"],"Accept-Encoding":["gzip, deflate, br"],"Pragma":["no-cache"],"Cache-Control":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"],"Upgrade":["websocket"],"Accept-Language":["en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"],"Sec-Websocket-Key":["y7UKeybqik9+96yfSE/bYQ=="],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"bitwarden.my.server"}},"duration":0.000908604,"status":502,"err_id":"94h4wk8fi","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:440)"}
Oct 01 14:42:31 my-server caddy[26985]: {"level":"error","ts":1601552551.7421207,"logger":"http.log.error","msg":"read tcp 127.0.0.1:50416->127.0.0.1:3012: read: connection reset by peer","request":{"remote_addr":"79.177.86.25:60278","proto":"HTTP/1.1","method":"GET","host":"bitwarden.my.server","uri":"/notifications/hub?access_token=%VERYLONGTOKEN%","headers":{"Connection":["Upgrade"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"],"Sec-Websocket-Key":["XEx+bGxiJSNEUSCGahoWXg=="],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Accept-Language":["en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"],"Pragma":["no-cache"],"Cache-Control":["no-cache"],"Upgrade":["websocket"],"Origin":["chrome-extension://nngceckbapebfimnlniiiahkandclblb"],"Sec-Websocket-Version":["13"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"bitwarden.my.server"}},"duration":0.000882846,"status":502,"err_id":"xm3r8hjw9","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:440)"}
Oct 01 14:44:40 my-server caddy[26985]: {"level":"error","ts":1601552680.6990004,"logger":"http.log.error","msg":"read tcp 127.0.0.1:50642->127.0.0.1:3012: read: connection reset by peer","request":{"remote_addr":"79.177.86.25:60286","proto":"HTTP/1.1","method":"GET","host":"bitwarden.my.server","uri":"/notifications/hub?access_token=%VERYLONGTOKEN%","headers":{"Sec-Websocket-Version":["13"],"Sec-Websocket-Key":["Arjw5QTHshL0jXdLvouKJw=="],"Origin":["chrome-extension://nngceckbapebfimnlniiiahkandclblb"],"Cache-Control":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"],"Upgrade":["websocket"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Connection":["Upgrade"],"Pragma":["no-cache"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"bitwarden.my.server"}},"duration":0.000834268,"status":502,"err_id":"6d8ajs7hu","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:440)"}
Oct 01 17:07:55 my-server caddy[26985]: {"level":"error","ts":1601561275.4082503,"logger":"http.log.error","msg":"read tcp 127.0.0.1:48700->127.0.0.1:3012: read: connection reset by peer","request":{"remote_addr":"79.177.86.25:60433","proto":"HTTP/1.1","method":"GET","host":"bitwarden.my.server","uri":"/notifications/hub?access_token=%VERYLONGTOKEN%","headers":{"Accept-Language":["en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"],"Connection":["Upgrade"],"Pragma":["no-cache"],"Sec-Websocket-Version":["13"],"Upgrade":["websocket"],"Origin":["chrome-extension://nngceckbapebfimnlniiiahkandclblb"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Websocket-Key":["l4L2VEhNd53SdO+RVZXM3g=="],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Cache-Control":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"bitwarden.my.server"}},"duration":0.010254758,"status":502,"err_id":"nsk41fjnw","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:440)"}
Oct 01 17:11:01 my-server caddy[26985]: {"level":"error","ts":1601561461.8718193,"logger":"http.log.error","msg":"read tcp 127.0.0.1:49258->127.0.0.1:3012: read: connection reset by peer","request":{"remote_addr":"79.177.86.25:60489","proto":"HTTP/1.1","method":"GET","host":"bitwarden.my.server","uri":"/notifications/hub?access_token=%VERYLONGTOKEN%","headers":{"Upgrade":["websocket"],"Sec-Websocket-Key":["Gn0ws9hnT9JqM/SsAwJ+tg=="],"Cache-Control":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"],"Sec-Websocket-Version":["13"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Connection":["Upgrade"],"Pragma":["no-cache"],"Origin":["chrome-extension://nngceckbapebfimnlniiiahkandclblb"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"bitwarden.my.server"}},"duration":0.000818097,"status":502,"err_id":"rns5qeqnr","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:440)"}
Oct 01 17:15:21 my-server caddy[26985]: {"level":"error","ts":1601561721.2472804,"logger":"http.log.error","msg":"read tcp 127.0.0.1:49858->127.0.0.1:3012: read: connection reset by peer","request":{"remote_addr":"79.177.86.25:60494","proto":"HTTP/1.1","method":"GET","host":"bitwarden.my.server","uri":"/notifications/hub?access_token=%VERYLONGTOKEN%","headers":{"Accept-Language":["en-US,en;q=0.9,he-IL;q=0.8,he;q=0.7"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Cache-Control":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"],"Origin":["chrome-extension://nngceckbapebfimnlniiiahkandclblb"],"Upgrade":["websocket"],"Sec-Websocket-Version":["13"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Websocket-Key":["PmeBNQ+65EHot2Pr/t9d2g=="],"Connection":["Upgrade"],"Pragma":["no-cache"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"bitwarden.my.server"}},"duration":0.000964056,"status":502,"err_id":"n2z9g611r","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:440)"}

I have also made sure that caddy and my router setup does handle websocket connections properly by using a simple websocket server and accessing it remotely, so it must be something with my specific bitwarden or caddy configuration.

Bitwarden docker-compose (basically the same from bitwarden_rs github):

#---
#Docker-compose file for Bitwarden_rs
#--- 
version: "3"
services: 
  bitwardenrs: 
    restart: always
    # Dani Garcia image https://github.com/dani-garcia/bitwarden_rs
    image: "bitwardenrs/server:latest"
    container_name: bitwardenrs
    environment:
      # Logging connection attemps
      - LOG_FILE=/data/bitwarden.log
      - EXTENDED_LOGGING='true'
      - LOG_LEVEL=warn
      - WEBSOCKET_ENABLED='true'
      - SIGNUPS_ALLOWED='true'
      #- DISABLE_ADMIN_TOKEN='true'
      #- ADMIN_TOKEN=YouRandomTokenHere
      - SHOW_PASSWORD_HINT='true'
      - DISABLE_ICON_DOWNLOAD='true'
      #- SMTP_HOST=smtphost
      #- SMTP_PORT=port
      #- SMTP_SSL='true'
      #- SMTP_FROM=address_from@domain.tld
      #- SMTP_USERNAME=smtp_user_name
      #- SMTP_PASSWORD=smtp_password
    ports:
      - "3011:80"
      - "3012:3012"
    volumes: 
      - ./bw-data:/data

And my Caddyfile:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
{
### DEBUG CONFIGS
#    debug
#    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
###
    email   mymail@gmail.com
}
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace the line below with your
# domain name.

test.my.server {
#    respond "ZA WARUDO"
}

bitwarden.my.server {
    # log {
    #     output file {env.LOG_FILE}
    #     level INFO
    #     #roll_size 5MiB #Not working on Caddy V2.0.0 Beta20 https://caddyserver.com/docs/caddyfile/directives/log#log
    #     #rool_keep 30 #Not working on Caddy V2.0.0 Beta20 https://caddyserver.com/docs/caddyfile/directives/log#log
    # }

    encode gzip

    header / {
        # Enable HTTP Strict Transport Security (HSTS)
        Strict-Transport-Security "max-age=31536000;"
        # Enable cross-site filter (XSS) and tell browser to block detected attacks
        X-XSS-Protection "1; mode=block"
        # Disallow the site to be rendered within a frame (clickjacking protection)
        X-Frame-Options "DENY"
        # Prevent search engines from indexing (optional)
        X-Robots-Tag "none"
        # Server name removing
        -Server
    }
    # The negotiation endpoint is also proxied to Rocket
    reverse_proxy /notifications/hub/negotiate http://localhost:3011  
    # Notifications redirected to the websockets server
    reverse_proxy /notifications/hub http://localhost:3012  
    # Proxy the Root directory to Rocket
    reverse_proxy http://localhost:3011 {
         # Send the true remote IP to Rocket, so that bitwarden_rs can put this in the
         # log, so that fail2ban can ban the correct IP.
         header_up X-Real-IP {remote_host}
    }
}

Thanks, any help would be extremely appriciated :slight_smile: