504 Gateway Time-out when importing vault

I’ve got a fresh install of Vaultwarden in a docker container. Install script below:

docker pull vaultwarden/server:latest
docker run -d --name vaultwarden -v /vw-data/:/data/ --restart unless-stopped -p 8080:80 vaultwarden/server:latest

I’m using npm on a different machine for reverse proxy and lets encrypt cert. npm config below:

I can login and everything seems to be working fine, however when I attempt to import my vault I get a 504 Gateway Time-out error.

If I only place a couple lines in the “or copy/paste the import file contents” box, they data will import, but anything over a few lines just gives the same 504 error.

This is what I see in my npm error log:

2024/03/07 23:13:57 [warn] 1357#1357: *15349 a client request body is buffered to a temporary file /tmp/nginx/body/6/16/0000000166, client: [CLIENTIPADDR], server: [SVRDNSNAME], request: "POST /api/ciphers/import HTTP/1.1", host: "[SVRDNSNAME]", referrer: "https://[SVRDNSNAME]/"

2024/03/07 23:15:27 [error] 1357#1357: *15349 upstream timed out (110: Connection timed out) while reading response header from upstream, client: [CLIENTIPADDR], server: [SVRDNSNAME], request: "POST /api/ciphers/import HTTP/1.1", upstream: "http://[SVRIPADDR]:8080/api/ciphers/import", host: "[SVRDNSNAME]", referrer: "https://[SVRDNSNAME]/"

I’ve attempted to change the network on the docker container to the host instead of the bridge just to make sure there wasn’t something with it not being able to communicate which didn’t help.

I added the following to npm to see if it helped and it didn’t:

location /admin {
  return 404;
  }

  location / {
    proxy_pass http://[SVRIPADDR]:8080;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
  
  location /notifications/hub {
    proxy_pass http://[SVRIPADDR]:3012;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }
  
  location /notifications/hub/negotiate {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass http://[SVRIPADDR]:8080;
  }

I even attempted to add some crazy timeouts to see if that made a difference and while I no longer got the error (because I didn’t wait that long) it also never imported.

proxy_connect_timeout 3000;
proxy_send_timeout 3000;
proxy_read_timeout 3000;
send_timeout 3000;

Depending if all data was delivered correctly to Vaultwarden, it should still be imported, just not directly visible.

Unless there are some invalid import items which you can’t see in the interface now, but should at least report some error in the Vaultwarden logs.

Only way to solve it is increasing the timeout of the reverse proxy.
Not sure which setting that is from the top of my head though.

I have my vaultwarden logs set to debug and there is nothing of note. no errors or warns

I’ve increased the timeouts in npm but that doesn’t fix the issue.

I suppose there could be some malformed entries but with over 700 entries, that’s more than a chore. Also, not knowing what would be considered malformed would make that task almost impossible.


I have roughly the same setup with my current installation, only differences are that nginx and vaultwarden are on the same server and it’s standard nginx and not npm. It could be an issue with npm and vaultwarden or the fact that they are not on the same server. I’m just not sure.

Start with verifying if the import request is being delivered to Vaultwarden at all. If not, then it’s something in front of it.

If the request gets delivered, but no errors, then the entries should be there.

Ah, ok. I’m not seeing anything in the logs to indicate that the import request is even making it to vaultwarden. But from this error, I feel like i should see something in the vaultwarden logs.

image

Looks more like a reverse proxy error, not Vaultwarden.
Not sure how large your final import is in the end, that which is send to Vaultwarden. But either your reverse proxy is having issues handling the load, or it’s not able to pass it through to Vaultwarden.

Try to enable debug logging for Vaultwarden and see if any requests are coming in regarding the import. If not, then the reverse proxy isn’t able to send data for some reason.

I did a reboot of the server so I could get a nice clear log of the entire process from the vaultwarden side.

[2024-03-08 13:56:14.682][vaultwarden::api::identity][INFO] User [USERNAME] logged in successfully. IP: [CLIENTIP]
[2024-03-08 13:56:14.682][response][INFO] (login) POST /identity/connect/token => 200 OK
[2024-03-08 13:56:14.710][request][INFO] GET /api/config
[2024-03-08 13:56:14.711][response][INFO] (config) GET /api/config => 200 OK
[2024-03-08 13:56:14.786][request][INFO] GET /images/icon-white.png
[2024-03-08 13:56:14.786][response][INFO] (web_files) GET /<p..> [10] => 200 OK
[2024-03-08 13:56:14.804][request][INFO] POST /identity/connect/token
[2024-03-08 13:56:14.806][response][INFO] (login) POST /identity/connect/token => 200 OK
[2024-03-08 13:56:14.821][request][INFO] GET /images/close-button.svg
[2024-03-08 13:56:14.821][response][INFO] (web_files) GET /<p..> [10] => 200 OK
[2024-03-08 13:56:14.844][request][INFO] GET /notifications/hub?access_token=eyJ0eXAiOiJKV1QiL
[2024-03-08 13:56:14.844][vaultwarden::api::notifications][INFO] Accepting Rocket WS connection from [CLIENTIP]
[2024-03-08 13:56:14.844][response][INFO] (websockets_hub) GET /notifications/hub?<data..> => 200 OK
[2024-03-08 13:56:14.850][request][INFO] GET /fonts/Open_Sans-normal-700.a39de6b1b80c7f21cf88.woff
[2024-03-08 13:56:14.850][response][INFO] (web_files) GET /<p..> [10] => 200 OK
[2024-03-08 13:56:14.861][request][INFO] GET /api/sync?excludeDomains=true
[2024-03-08 13:56:14.867][response][INFO] (sync) GET /api/sync?<data..> => 200 OK
[2024-03-08 13:56:14.890][request][INFO] GET /api/config
[2024-03-08 13:56:14.890][response][INFO] (config) GET /api/config => 200 OK
[2024-03-08 13:56:18.364][request][INFO] GET /395.38a9c38835d7b081bc1d.js
[2024-03-08 13:56:18.364][response][INFO] (web_files) GET /<p..> [10] => 200 OK
[2024-03-08 13:56:18.364][request][INFO] GET /677.f099bcd59898015da3cf.js
[2024-03-08 13:56:18.365][response][INFO] (web_files) GET /<p..> [10] => 200 OK
[2024-03-08 13:56:22.879][request][INFO] GET /alive
[2024-03-08 13:56:22.880][response][INFO] (alive) GET /alive => 200 OK
[2024-03-08 13:56:26.337][request][INFO] POST /api/ciphers/import
[2024-03-08 13:56:52.898][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
[2024-03-08 13:56:52.898][vaultwarden::api::core::accounts][DEBUG] Purging auth requests
[2024-03-08 13:57:15.962][request][INFO] GET /notifications/hub?access_token=eyJ0eXAiOiJKV1QiL
[2024-03-08 13:57:15.963][vaultwarden::api::notifications][INFO] Accepting Rocket WS connection from [CLIENTIP]
[2024-03-08 13:57:15.963][response][INFO] (websockets_hub) GET /notifications/hub?<data..> => 200 OK
[2024-03-08 13:57:22.947][request][INFO] GET /alive
[2024-03-08 13:57:22.948][response][INFO] (alive) GET /alive => 200 OK
[2024-03-08 13:57:52.901][vaultwarden::api::core::accounts][DEBUG] Purging auth requests
[2024-03-08 13:57:52.901][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
[2024-03-08 13:58:23.043][request][INFO] GET /alive
[2024-03-08 13:58:23.043][response][INFO] (alive) GET /alive => 200 OK
[2024-03-08 13:58:52.903][vaultwarden::api::core::accounts][DEBUG] Purging auth requests
[2024-03-08 13:58:52.903][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins

Accompanying npm log

2024/03/08 13:56:26 [warn] 1774#1774: *17062 a client request body is buffered to a temporary file /tmp/nginx/body/3/23/0000000233, client: [CLIENTIP], server: [SVRDNSNAME], request: "POST /api/ciphers/import HTTP/1.1", host: "[SVRDNSNAME]", referrer: "https://[SVRDNSNAME].com/"
2024/03/08 13:57:56 [error] 1774#1774: *17062 upstream timed out (110: Connection timed out) while reading response header from upstream, client: [CLIENTIP], server: [SRVDNSNAME], request: "POST /api/ciphers/import HTTP/1.1", upstream: "http://[SVRIPADDR]:8080/api/ciphers/import", host: "[SVRDNSNAME]", referrer: "https://[SVRDNSNAME]/"

This is a nginx config issue as nothing was passed to Vaultwarden.

You should disable proxy buffering and also enable http1.1.

proxy_request_buffering off;
proxy_http_version 1.1;

Also, you are using an outdated config for websocket connections. That will be removed in 1.31.

See ⚠️ [NOTE] BREAKING CHANGE in upcoming v1.31.0 stable version - #3

Just in case someone stumbles across this problem in the future, I finally figured out the issue.

This has nothing to do with Vaultwarden or even NPM this has to do with a firewall setting in OPNsense or PFsense. This is triggered when you have docker containers talking to each other across different servers. See link on the OPNsense forums. https://forum.opnsense.org/index.php?topic=18731.0