X-Forwarded-For issue

Hi there,

I am a brand new user of Bitwarden_rs and I really like it so far.
My setup is:
- pfSense with HAProxy and Let’s Encrypt SSL certificates
- 2 Frontends (1 for WAN, 1 for LAN) so I use the same URLs internally and externally
- Bitwarden_rs (1.14.2) in Docker (as per https://github.com/dani-garcia/bitwarden_rs/wiki/Starting-a-Container), on a Debian 10 VM
- Admin Token provided can configured
- SMTP working
- Fail2ban

Everything seems to work fine (I have not tried the notifications) but I am stuck with an issue on the client IP identification in Bitwarden_rs, and thus, in Fail2ban.

HAProxy uses the “X-Forwarded-For” option to provide the initial client’ IP address but, for some reason, Bitwarden_rs’ log always provide the pfSense’s IP address as the source.
I have tried with -e IP_HEADER=X-Forwarded-For and "-e IP_HEADER=X-Forwarded-For" when using “docker run”, but it doesn’t seem to change anything. And I don’t see any field in config.json that could counteract this environment variable.

I increased the log level to trace and I can see that the header has some IP address(es?) in the “X-Forwarded-For” field but I have no idea how to convert the raw text into real human readable IP address. Could somebody point me in the right direction to read this field?
All I can see so far it that the raw data is not identical if I connect from different devices (internally and externally) so I would expect the field to be correct.

And of course, I would appreciate it if someone who has managed to make it work using the “X-Forwarded-For” field could give me the setup they put in place to solve this.

Thank you in advance for your support,
Brice

I use HAProxy behind Cloudflare, and receiving the original client IP works fine for me (except with Websocket connections).

Make sure you have option forwardfor set in HAProxy. Otherwise, I’d guess your pfSense isn’t passing an X-Forwarded-For header, though I don’t know know anything about pfSense.

Thanks for your feedback. Did you use the Docker container or did you build from binaries?

The option is set on both Frontends. To make sure since I don’t know how to read the logs, I installed tshark and used tshark -i docker0 -O http -f "tcp port 80" -Y "http.request". This allowed me to read the headers coming in from HAProxy into Bitwarden_rs.
And I could see that the X-Forwarded-For field is correctly set with only one IP address: the client’s IP address.
So for me, HAProxy seems to work and be configured correctly.

My investigation leads back to Bitwarden_rs. Is there a way to know if the environment variable for IP_HEADER is correctly set and understood? I used with quotes and with double-quotes. Is there another way to try?

@brice_bourou, it should be just the text, without any type of quotes.
You can also try to increase the log_level to trace which will output the headers received.

So I did more digging and found out that there was a ip_header line that appeared in the config.json file. It bypassed my environment settings (obviously) so it couldn’t work.
I modified this value to X-Forwarded-Forand recreated the docker (in case) => still not working correctly.
I reset this value to X-Real-IP and changed my HAProxy configuration to modify this field in the request: it works as expected now.

Checking the headers with tshark showed me that the only difference between the 2 fields was that X-Forwarded-Foris ended by \r\n. It could be a reason for the parsing to fail and to revert to the X-Real-IP.

Anyway, my issue seems to be solved but I think there is a bug in Bitwarden_rs. Hope it will be solved :slight_smile:

@BlackDex I just saw your message. Thanks!

I already increased the log_level to trace but the headers received show the IP addresses in a strange format (to me at least), and I am not able to convert them back to a readable IP address.

I See that you say that there is a different value in the config.json.
That means that you have changed this value (or any other value for that matter) via the /admin interface.
So there are two ways of fixing this.

  1. Login to the /admin interface, change the value there.
  2. Remove the config.json file and check that all settings are correct in the environment variables, then restart.

I did not see any field in the admin page that you allow me to change this field so I changed it manually in the config.json file. I stop/rm/run/start’ed and it was still not working.

When I say “I modified this value” / “I reset this value”, I meant, in the config.json file. Sorry for not being specific enough.

After looking at the long list of trace log lines, I did not see anywhere where it could state which IP address was considered for the client IP.

So I went and modified my reverse proxy.