Hi there,
I’ve had my self host vaultwarden and caddy servers running for 3+ months but suddenly starting 29th October I am unsure which server stops responding properly. I am guessing its caddy but can’t be sure. The error is “Secure Connection Failed” and the error code is SSL_ERROR_INTERNAL_ERROR_ALERT.
To see clearly the log, I restarted docker-compose w/o detach and here it is:
someone@xubuntu1:~/MyDocker/vaultwarden$ docker-compose up
Creating network "vaultwarden_default" with the default driver
Creating caddy ... done
Creating bitwarden ... done
Attaching to caddy, bitwarden
bitwarden | /--------------------------------------------------------------------\
bitwarden | | Starting Vaultwarden |
bitwarden | | Version 1.23.0 |
bitwarden | |--------------------------------------------------------------------|
bitwarden | | This is an *unofficial* Bitwarden implementation, DO NOT use the |
bitwarden | | official channels to report bugs/features, regardless of client. |
bitwarden | | Send usage/configuration questions or feature requests to: |
bitwarden | | https://vaultwarden.discourse.group/ |
bitwarden | | Report suspected bugs/issues in the software itself at: |
bitwarden | | https://github.com/dani-garcia/vaultwarden/issues/new |
bitwarden | \--------------------------------------------------------------------/
bitwarden |
bitwarden | [INFO] No .env file found.
bitwarden |
bitwarden | [2021-10-31 16:30:44.898][parity_ws][INFO] Listening for new connections on 0.0.0.0:3012.
caddy | {"level":"info","ts":1635712244.7120955,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
bitwarden | [2021-10-31 16:30:44.899][start][INFO] Rocket has launched from http://0.0.0.0:8080
caddy | {"level":"warn","ts":1635712244.7143478,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddy | {"level":"info","ts":1635712244.7160575,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
caddy | {"level":"info","ts":1635712244.7166455,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy | {"level":"info","ts":1635712244.7166684,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy | {"level":"info","ts":1635712244.7172618,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00033df80"}
caddy | {"level":"info","ts":1635712244.718392,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["XXX.YYY.org"]}
caddy | {"level":"info","ts":1635712244.7188015,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy | {"level":"info","ts":1635712244.7188163,"msg":"serving initial configuration"}
caddy | {"level":"info","ts":1635712244.719158,"logger":"tls.obtain","msg":"acquiring lock","identifier":"XXX.YYY.org"}
caddy | {"level":"info","ts":1635712244.7193525,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy | {"level":"info","ts":1635712244.719702,"logger":"tls","msg":"finished cleaning storage units"}
caddy | {"level":"info","ts":1635712244.8648643,"logger":"tls.obtain","msg":"lock acquired","identifier":"XXX.YYY.org"}
caddy | {"level":"info","ts":1635712244.8886557,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["XXX.YYY.org"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
caddy | {"level":"info","ts":1635712244.888683,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["XXX.YYY.org"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
caddy | {"level":"warn","ts":1635712254.8936892,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:33476->127.0.0.11:53: i/o timeout"}
caddy | {"level":"warn","ts":1635712265.1452632,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:35217->127.0.0.11:53: i/o timeout"}
caddy | {"level":"warn","ts":1635712275.399445,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:41118->127.0.0.11:53: i/o timeout"}
caddy | {"level":"error","ts":1635712275.3995533,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"XXX.YYY.org","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[XXX.YYY.org] creating new order: provisioning client: performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:41118->127.0.0.11:53: i/o timeout (ca=https://acme-v02.api.letsencrypt.org/directory)"}
caddy | {"level":"warn","ts":1635712275.3998773,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
caddy | {"level":"error","ts":1635712285.4019375,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"XXX.YYY.org","issuer":"acme.zerossl.com-v2-DV90","error":"account pre-registration callback: performing EAB credentials request: Post \"https://api.zerossl.com/acme/eab-credentials-email\": dial tcp: lookup api.zerossl.com on 127.0.0.11:53: read udp 127.0.0.1:32920->127.0.0.11:53: i/o timeout"}
caddy | {"level":"error","ts":1635712285.4019938,"logger":"tls.obtain","msg":"will retry","error":"[XXX.YYY.org] Obtain: account pre-registration callback: performing EAB credentials request: Post \"https://api.zerossl.com/acme/eab-credentials-email\": dial tcp: lookup api.zerossl.com on 127.0.0.11:53: read udp 127.0.0.1:32920->127.0.0.11:53: i/o timeout","attempt":1,"retrying_in":60,"elapsed":40.537078263,"max_duration":2592000}
caddy | {"level":"warn","ts":1635712355.4173844,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:42370->127.0.0.11:53: i/o timeout"}
caddy | {"level":"warn","ts":1635712365.6722748,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:38577->127.0.0.11:53: i/o timeout"}
caddy | {"level":"warn","ts":1635712375.924883,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:56576->127.0.0.11:53: i/o timeout"}
caddy | {"level":"error","ts":1635712375.9250164,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"XXX.YYY.org","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[XXX.YYY.org] creating new order: provisioning client: performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:56576->127.0.0.11:53: i/o timeout (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
caddy | {"level":"warn","ts":1635712375.925339,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
caddy | {"level":"error","ts":1635712385.930896,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"XXX.YYY.org","issuer":"acme.zerossl.com-v2-DV90","error":"account pre-registration callback: performing EAB credentials request: Post \"https://api.zerossl.com/acme/eab-credentials-email\": dial tcp: lookup api.zerossl.com on 127.0.0.11:53: read udp 127.0.0.1:37220->127.0.0.11:53: i/o timeout"}
caddy | {"level":"error","ts":1635712385.9309554,"logger":"tls.obtain","msg":"will retry","error":"[XXX.YYY.org] Obtain: account pre-registration callback: performing EAB credentials request: Post \"https://api.zerossl.com/acme/eab-credentials-email\": dial tcp: lookup api.zerossl.com on 127.0.0.11:53: read udp 127.0.0.1:37220->127.0.0.11:53: i/o timeout","attempt":2,"retrying_in":120,"elapsed":141.066039083,"max_duration":2592000}
Other than normal updates applied thru “apt update”, I don’t recall doing any changes since both servers are up and running (if it aint broken, dont fix it). The update to v1.23.0 is done after things are broken, thinking that update to latest would help.
Further examining the above log, it seems like caddy failed to obtain certificate from issuer after invoking acme-v02.api.letsencrypt.org-directory. I don’t remember I have done anything special at let’s encrypt but I could be wrong. Also,
- why are there 127.0.0.11? I don’t have anything setup for that.
- zerossl said they need an email address. But how do I do that?
Can some gurus help? Could there be change at let’s encrypt side that I need to sync up? Now I couldn’t connect to my bitwarden server so I cannot add/update password no more.