My setup is as follows:
I run an Archlinux LXC container on Proxmox and the only service that is run in this container is vaultwarden. The SSL is handled by HAProxy plugin installed on my OpnSense firewall
I use the Arch packages – vaultwarden & vaultwarden-web from the community repository
I updated my LXC container and noticed that the vaultwarden-web package had an update and so I updated to vaultwarden-web-2022.9.2-1. After that I thought of taking a backup of the container, so I shutdown the container and took the backup. But after reboot, I have been unable to access the web-vault.
No changes have been made to the HAProxy configuration in many months. All I get in the browser is 503 Service Unavailable.
Relevant entries in the /etc/vaultwarden.env file (I have created a symlink in the /var/lib/vaultwarden/.env just to avoid the “No .env file found” error in the logs)
DATA_FOLDER=/var/lib/vaultwarden
WEB_VAULT_FOLDER=/usr/share/webapps/vaultwarden-web
WEB_VAULT_ENABLED=true
LOG_FILE=/var/log/vaultwarden.log
LOG_LEVEL=InfoWhen I have the log level as info, the only logs I get in the /var/log/vaultwarden is
[2022-09-16 04:01:19.123][start][INFO] Rocket has launched from 127.0.0.1:8000and checking the status via systemd gives me
vaultwarden.service - Vaultwarden Server
Loaded: loaded (/usr/lib/systemd/system/vaultwarden.service; enabled; preset: disabled)
Active: active (running) since Fri 2022-09-16 04:01:19 UTC; 11h ago
Docs:github.com/dani-garcia/vaultwarden
Main PID: 996 (vaultwarden)
Tasks: 9 (limit: 38403)
Memory: 8.3M
CPU: 720ms
CGroup: /system.slice/vaultwarden.service
└─996 /usr/bin/vaultwarden
Sep 16 04:01:19 vaultwarden vaultwarden[996]: | Version 1.25.2 |
Sep 16 04:01:19 vaultwarden vaultwarden[996]: |--------------------------------------------------------------------|
Sep 16 04:01:19 vaultwarden vaultwarden[996]: | This is an *unofficial* Bitwarden implementation, DO NOT use the |
Sep 16 04:01:19 vaultwarden vaultwarden[996]: | official channels to report bugs/features, regardless of client. |
Sep 16 04:01:19 vaultwarden vaultwarden[996]: | Send usage/configuration questions or feature requests to: |
Sep 16 04:01:19 vaultwarden vaultwarden[996]: | vaultwarden.discourse.group/ |
Sep 16 04:01:19 vaultwarden vaultwarden[996]: | Report suspected bugs/issues in the software itself at: |
Sep 16 04:01:19 vaultwarden vaultwarden[996]: | github.com/dani-garcia/vaultwarden/issues/new |
Sep 16 04:01:19 vaultwarden vaultwarden[996]: \--------------------------------------------------------------------/
Sep 16 04:01:19 vaultwarden vaultwarden[996]: [2022-09-16 04:01:19.123][start][INFO] Rocket has launched from 127.0.0.1:8000when setting the log level to debug I get
152 [2022-09-16 15:20:54.213][routes][INFO] GET /api/plans/sales-tax-rates
153 [2022-09-16 15:20:54.213][routes][INFO] POST /api/sends
154 [2022-09-16 15:20:54.213][routes][INFO] GET /api/sends
155 [2022-09-16 15:20:54.213][routes][INFO] PUT /api/sends/<id>
156 [2022-09-16 15:20:54.213][routes][INFO] DELETE /api/sends/<id>
157 [2022-09-16 15:20:54.213][routes][INFO] PUT /api/sends/<id>/remove-password
158 [2022-09-16 15:20:54.213][routes][INFO] GET /api/sends/<send_id>/<file_id>?<t>
159 [2022-09-16 15:20:54.213][routes][INFO] POST /api/sends/<send_id>/access/file/<file_id>
160 [2022-09-16 15:20:54.213][routes][INFO] GET /api/sends/<uuid>
161 [2022-09-16 15:20:54.213][routes][INFO] POST /api/sends/access/<access_id>
162 [2022-09-16 15:20:54.213][routes][INFO] POST /api/sends/file
163 [2022-09-16 15:20:54.213][routes][INFO] POST /api/settings/domains
164 [2022-09-16 15:20:54.213][routes][INFO] GET /api/settings/domains
165 [2022-09-16 15:20:54.213][routes][INFO] PUT /api/settings/domains
166 [2022-09-16 15:20:54.213][routes][INFO] GET /api/sync?<data..>
167 [2022-09-16 15:20:54.213][routes][INFO] GET /api/two-factor
168 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/authenticator
169 [2022-09-16 15:20:54.213][routes][INFO] PUT /api/two-factor/authenticator
170 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/disable
171 [2022-09-16 15:20:54.213][routes][INFO] PUT /api/two-factor/disable
172 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/duo
173 [2022-09-16 15:20:54.213][routes][INFO] PUT /api/two-factor/duo
174 [2022-09-16 15:20:54.213][routes][INFO] PUT /api/two-factor/email
175 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/get-authenticator
176 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/get-duo
177 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/get-email
178 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/get-recover
179 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/get-webauthn
180 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/get-webauthn-challenge
181 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/get-yubikey
182 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/recover
183 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/send-email
184 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/send-email-login
185 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/webauthn
186 [2022-09-16 15:20:54.213][routes][INFO] PUT /api/two-factor/webauthn
187 [2022-09-16 15:20:54.213][routes][INFO] DELETE /api/two-factor/webauthn
188 [2022-09-16 15:20:54.213][routes][INFO] POST /api/two-factor/yubikey
189 [2022-09-16 15:20:54.213][routes][INFO] PUT /api/two-factor/yubikey
190 [2022-09-16 15:20:54.213][routes][INFO] GET /api/users/<uuid>/public-key
191 [2022-09-16 15:20:54.213][routes][INFO] GET /api/version
192 [2022-09-16 15:20:54.213][routes][INFO] GET /app-id.json
193 [2022-09-16 15:20:54.213][routes][INFO] GET /attachments/<uuid>/<file_id>
194 [2022-09-16 15:20:54.213][routes][INFO] GET /icons/<domain>/icon.png
195 [2022-09-16 15:20:54.213][routes][INFO] POST /identity/accounts/prelogin
196 [2022-09-16 15:20:54.213][routes][INFO] POST /identity/connect/token
197 [2022-09-16 15:20:54.213][routes][INFO] GET /notifications/hub
198 [2022-09-16 15:20:54.213][routes][INFO] POST /notifications/hub/negotiate
199 [2022-09-16 15:20:54.213][routes][INFO] GET /vw_static/<filename>
200 [2022-09-16 15:20:54.213][start][INFO] Rocket has launched from 127.0.0.1:8000
201 [2022-09-16 15:21:54.216][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
202 [2022-09-16 15:22:54.218][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
203 [2022-09-16 15:23:54.220][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
204 [2022-09-16 15:24:54.222][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
205 [2022-09-16 15:25:54.225][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
206 [2022-09-16 15:26:54.228][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
207 [2022-09-16 15:27:54.230][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
208 [2022-09-16 15:28:54.232][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
209 [2022-09-16 15:29:54.234][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
210 [2022-09-16 15:30:54.236][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
211 [2022-09-16 15:31:54.237][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
212 [2022-09-16 15:32:54.240][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
213 [2022-09-16 15:33:54.242][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
214 [2022-09-16 15:34:54.245][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
215 [2022-09-16 15:35:54.248][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
216 [2022-09-16 15:36:54.250][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
217 [2022-09-16 15:37:54.252][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
218 [2022-09-16 15:38:54.255][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
219 [2022-09-16 15:39:54.258][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
220 [2022-09-16 15:40:54.260][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
221 [2022-09-16 15:41:54.263][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
222 [2022-09-16 15:42:54.264][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
223 [2022-09-16 15:43:54.267][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
224 [2022-09-16 15:44:54.269][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
225 [2022-09-16 15:45:54.272][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
226 [2022-09-16 15:46:54.275][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
227 [2022-09-16 15:47:54.276][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
228 [2022-09-16 15:48:54.279][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
229 [2022-09-16 15:49:54.281][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
230 [2022-09-16 15:50:54.284][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
231 [2022-09-16 15:51:54.287][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
232 [2022-09-16 15:52:54.289][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
233 [2022-09-16 15:53:54.291][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
234 [2022-09-16 15:54:54.295][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
235 [2022-09-16 15:55:54.297][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
236 [2022-09-16 15:56:54.299][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
237 [2022-09-16 15:57:54.300][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
238 [2022-09-16 15:58:54.302][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
239 [2022-09-16 15:59:54.310][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
240 [2022-09-16 16:00:54.313][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
241 [2022-09-16 16:01:54.314][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
242 [2022-09-16 16:02:54.317][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
305 [2022-09-16 17:00:54.472][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
306 [2022-09-16 17:01:54.475][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
307 [2022-09-16 17:02:54.476][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
308 [2022-09-16 17:03:54.479][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
309 [2022-09-16 17:04:54.481][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
310 [2022-09-16 17:05:24.485][vaultwarden::api::core::sends][DEBUG] Purging sends
311 [2022-09-16 17:05:24.485][vaultwarden::api::core::emergency_access][DEBUG] Start emergency_request_timeout_job
312 [2022-09-16 17:05:24.485][vaultwarden::api::core::emergency_access][DEBUG] Start emergency_notification_reminder_job
313 [2022-09-16 17:05:24.486][vaultwarden::api::core::emergency_access][DEBUG] No emergency request timeout to approve
314 [2022-09-16 17:05:24.486][vaultwarden::api::core::emergency_access][DEBUG] No emergency request reminder notification to send
315 [2022-09-16 17:05:54.486][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
316 [2022-09-16 17:06:54.488][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
317 [2022-09-16 17:07:54.490][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
318 [2022-09-16 17:08:54.493][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
319 [2022-09-16 17:09:54.494][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
320 [2022-09-16 17:10:54.496][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
321 [2022-09-16 17:11:54.499][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
322 [2022-09-16 17:12:54.500][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
323 [2022-09-16 17:13:54.503][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
324 [2022-09-16 17:14:54.504][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
325 [2022-09-16 17:15:54.507][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
326 [2022-09-16 17:16:54.509][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
327 [2022-09-16 17:17:54.510][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
328 [2022-09-16 17:18:54.513][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
329 [2022-09-16 17:19:54.516][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
330 [2022-09-16 17:20:54.518][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
331 [2022-09-16 17:21:54.520][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
332 [2022-09-16 17:22:54.524][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
333 [2022-09-16 17:23:54.526][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
334 [2022-09-16 17:24:54.530][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA loginsand the systemd status shows:
vaultwarden.service - Vaultwarden Server
Loaded: loaded (/usr/lib/systemd/system/vaultwarden.service; enabled; preset: disabled)
Active: active (running) since Fri 2022-09-16 15:20:54 UTC; 54s ago
Docs: github.com/dani-garcia/vaultwarden
Main PID: 2011 (vaultwarden)
Tasks: 8 (limit: 38403)
Memory: 11.6M
CPU: 70ms
CGroup: /system.slice/vaultwarden.service
└─2011 /usr/bin/vaultwarden
Sep 16 15:20:54 vaultwarden vaultwarden[2011]: [2022-09-16 15:20:54.213][routes][INFO] GET /api/version
Sep 16 15:20:54 vaultwarden vaultwarden[2011]: [2022-09-16 15:20:54.213][routes][INFO] GET /app-id.json
Sep 16 15:20:54 vaultwarden vaultwarden[2011]: [2022-09-16 15:20:54.213][routes][INFO] GET /attachments/<uuid>/<file_id>
Sep 16 15:20:54 vaultwarden vaultwarden[2011]: [2022-09-16 15:20:54.213][routes][INFO] GET /icons/<domain>/icon.png
Sep 16 15:20:54 vaultwarden vaultwarden[2011]: [2022-09-16 15:20:54.213][routes][INFO] POST /identity/accounts/prelogin
Sep 16 15:20:54 vaultwarden vaultwarden[2011]: [2022-09-16 15:20:54.213][routes][INFO] POST /identity/connect/token
Sep 16 15:20:54 vaultwarden vaultwarden[2011]: [2022-09-16 15:20:54.213][routes][INFO] GET /notifications/hub
Sep 16 15:20:54 vaultwarden vaultwarden[2011]: [2022-09-16 15:20:54.213][routes][INFO] POST /notifications/hub/negotiate
Sep 16 15:20:54 vaultwarden vaultwarden[2011]: [2022-09-16 15:20:54.213][routes][INFO] GET /vw_static/<filename>
Sep 16 15:20:54 vaultwarden vaultwarden[2011]: [2022-09-16 15:20:54.213][start][INFO] Rocket has launched from 127.0.0.1:8000But unfortunately, nothing in there shows me what is wrong and why the web-vault is inaccessible. Is there any way I can check whether vaultwarden & it’s web-vault is working correctly without worrying about the SSL part of it?
As a note: my HAProxy on Opnsense also is a reverse proxy/SSL offload for a bunch of other services like nextcloud, collabora, Omada, shinobi, transmission-web, emby, jellyfin etc etc. and they all seem to be working fine. Only vaultwarden-web is not – which leads me to believe it’s the vaultwarden container that is the issue and not the HAProxy configuration.
Can someone please help me figure this out?
Thanks in advance.