Issues with new 2023 image

I’ve (maybe) problems with the last update. I’m using podman on Alma 9 linux servers.

On my test server I updated the image - done this several times before.

[root@vaultwarden-test ~]# podman container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
74a588a8103e docker.io/vaultwarden/server:latest /start.sh 2 hours ago Up 2 hours ago 127.0.0.1:8080->80/tcp vaultwarden

My production server:

[root@vaultwarden ~]# podman container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ba7b12364767 docker.io/vaultwarden/server:latest /start.sh 4 months ago Up 4 weeks ago (healthy) 127.0.0.1:8080->80/tcp vaultwarden

The difference is the status on the updated server doesn’t indicate healthy.

I can login to the test server and us it.

GUI for vaultwarden-test is showing
Version 2023.3.0

GUI for vaultwarden is showing
Version 2022.12.0

inspectiong the image tor the test-machine:

“Health”: {
“Status”: “”,
“FailingStreak”: 0,
“Log”: null
},

The same om the production system:

“Health”: {
“Status”: “healthy”,
“FailingStreak”: 0,
“Log”: [
{
“Start”: “2023-05-11T08:47:06.086604482+02:00”,
“End”: “2023-05-11T08:47:06.198006045+02:00”,
“ExitCode”: 0,
“Output”: “”
},

So what is wrong - or is something wrong?

Some logs and config parameters would be useful

the podman log vaultwarden shows:

> [2023-05-12 10:33:59.967][start][INFO] Rocket has launched from http://0.0.0.0:80
> [2023-05-22 04:49:20.267][rocket::server][WARN] Received SIGTERM. Requesting shutdown.
> [2023-05-22 04:49:20.268][vaultwarden][INFO] Vaultwarden process exited!
> /--------------------------------------------------------------------\
> |                        Starting Vaultwarden                        |
> |                           Version 1.27.0                           |
> |--------------------------------------------------------------------|
> | This is an *unofficial* Bitwarden implementation, DO NOT use the   |
> | official channels to report bugs/features, regardless of client.   |
> | Send usage/configuration questions or feature requests to:         |
> |   https://vaultwarden.discourse.group/                             |
> | Report suspected bugs/issues in the software itself at:            |
> |   https://github.com/dani-garcia/vaultwarden/issues/new            |
> \--------------------------------------------------------------------/
> 
> [INFO] No .env file found.
> 
> [WARNING] The following environment variables are being overriden by the config file,
> [WARNING] please use the admin panel to make changes to them:
> [WARNING] ADMIN_TOKEN
> 
> [2023-05-22 04:50:09.282][start][INFO] Rocket has launched from http://0.0.0.0:80

Then after pull and installing the new image it is:

> /--------------------------------------------------------------------\
> |                        Starting Vaultwarden                        |
> |                           Version 1.28.1                           |
> |--------------------------------------------------------------------|
> | This is an *unofficial* Bitwarden implementation, DO NOT use the   |
> | official channels to report bugs/features, regardless of client.   |
> | Send usage/configuration questions or feature requests to:         |
> |   https://github.com/dani-garcia/vaultwarden/discussions or        |
> |   https://vaultwarden.discourse.group/                             |
> | Report suspected bugs/issues in the software itself at:            |
> |   https://github.com/dani-garcia/vaultwarden/issues/new            |
> \--------------------------------------------------------------------/
> 
> [INFO] Using saved config from `data/config.json` for configuration.
> 
> [NOTICE] You are using a plain text `ADMIN_TOKEN` which is insecure.
> Please generate a secure Argon2 PHC string by using `vaultwarden hash` or `argon2`.
> See: https://github.com/dani-garcia/vaultwarden/wiki/Enabling-admin-page#secure-the-admin_token
> 
> [2023-05-25 09:13:08.434][start][INFO] Rocket has launched from http://0.0.0.0:80

these are the podman log <container> outputs.

As said it is running and I can login at GUI but "healthy" is missing in status:

> [root@vaultwarden-test ~]# podman ps
> CONTAINER ID  IMAGE                                COMMAND     CREATED        STATUS        PORTS                   NAMES
> 4def8ec0252e  docker.io/vaultwarden/server:latest  /start.sh   7 seconds ago  Up 7 seconds  127.0.0.1:8080->80/tcp  vaultwarden

In which logs are you interested too?

So, what happens of you run the healthcheck.sh script from within the container? Because that script is what determines if it’s healthy or not.

before update:

[root@vaultwarden ~]# podman ps
CONTAINER ID  IMAGE                                COMMAND     CREATED       STATUS                  PORTS                   NAMES
ba7b12364767  docker.io/vaultwarden/server:latest  /start.sh   4 months ago  Up 3 minutes (healthy)  127.0.0.1:8080->80/tcp  vaultwarden
[root@vaultwarden ~]# podman healthcheck run vaultwarden
[root@vaultwarden ~]# podman exec -it vaultwarden sh healthcheck.sh
"2023-05-26T06:10:20.784489Z"[root@vaultwarden ~]# 

after update

[root@vaultwarden ~]# podman healthcheck run vaultwarden
Error: container 96bc7c7e9dd895281d96df1a61ae6ee31abb3bb91a496e77cf759768e1e61d91 has no defined healthcheck

but it is there:

[root@vaultwarden ~]# podman exec -it vaultwarden bash
root@ba7b12364767:/# ls
bin   dev	      home   media  proc  sbin	    sys  var
boot  etc	      lib    mnt    root  srv	    tmp  vaultwarden
data  healthcheck.sh  lib64  opt    run   start.sh  usr  web-vault
root@ba7b12364767:/# sh healthcheck.sh 
"2023-05-26T05:54:45.366760Z"root@ba7b12364767:/# 
root@ba7b12364767:/# exit   
exit

If I replace podman by docker-ce it is healthy:

[root@vaultwarden-test ~]# docker ps
CONTAINER ID   IMAGE                       COMMAND       CREATED        STATUS                  PORTS                              NAMES
507cd6da7a06   vaultwarden/server:latest   "/start.sh"   18 hours ago   Up 16 hours (healthy)   3012/tcp, 127.0.0.1:8080->80/tcp   vaultwarden

there is no docker healthcheck command

Differences getting the image

podman pull docker.io/vaultwarden/server
docker pull vaultwarden/server

Well, there definitely is a healthcheck defined in the images.
See Docker

So, maybe an issue with podman?
What if you try an image from quay.io or ghcr.io? Does that make a difference?

And also see vaultwarden/Dockerfile.j2 at 445680fb84d238aa432fdec271235933043410e4 · dani-garcia/vaultwarden · GitHub

I think it’s an issue with the OCI manifest v1. The healthcheck is only for docker. See this discussion in the image-spec repository: