Could you put the logging of bitwarden to debug or even trace and try again?
Right. I’ve enabled TRACE logs in the conf file and restarted the service.
I’ve repeated the following sequence of commands with user1:
bw login bw sync bw sync --last bw list items
And here’s the Bitwarden_rs log output:
https://pastebin.com/7Y0JZCtg
Strange enough that is only the token request, and nothing more.
I’m trying to figure out how we can troubleshoot shoot this without deleting every single vault item one at a time. But it’s getting very hard.
True… somehow this might get related to the fact that the bitwarden_rs service logs the successful sync attempts, and not the failed ones.
Whilst Nginx records both sync attempts: successful and failed.
In my mind it’s making sense the fact that Nginx is rejecting those sync attempts and it isn’t allowing that information to be known by bitwarden_rs service. Does it sound reasonable?
But I sincerely don’t know where to keep digging…
No, those requests should go to bitwarden, bitwarden should return a 4xx it crash or whatever.
So it then looks like nginx really isn’t passing them to bitwarden at all. One step further would be using tcpdump/Wireshark on the Ubuntu node to see what is coming in. That would be my next step in this case.
Hi, today I did some methodical testing to narrow down at which point a new user shifts from syncing to not syncing from CLI/Android App.
The conclusion is that when a user belongs to more than 1 organization, the syncing fails from CLI/Android App.
I have 3 organizations defined in the system. Whenever I enable access to a user to more than one organization, the sync fails.
And when I limit the access to only 1 organization, the syncing goes through.
For this to be visible, I had to login/logout back and forth at CLI everytime I updated the organization access. Otherwise, the syncing failure wouldn’t show up until the next login.
I’m under the impression that this error is system-wide, and you could replicate it too.
Because I’ve been consistently testing it with multiple combination of users and organizations, and the result doesn’t show any correlation between specific users/orgs as for them to be corrupted (somehow) in the database.
I don’t think that is the issue, since i have more then 3 orgs in my test environment.
What i find interesting is if those orgs are the same as you already have. If that is the case, it could be that something is wrong in that organization, either a cipher, or settings etc…
Can you confirm this?
What if you create 3 brand new organizations?
Hi, I’ve been testing as follows:
- I removed
user3from all organizations. At this point,user3can only access one random item stored in his personal vault. user1created 3 brand new organizations:OrgX,OrgYandOrgZ.user1created 1 random item inside each new organization:ItemX,ItemYandItemZ.user3was added toOrgX. At this pointuser3synced through CLI and got successful access toItemX.user3was removed fromOrgXand was added toOrgY. At this pointuser3synced through CLI and got successful access toItemY.user3was removed fromOrgYand was added toOrgZ. At this pointuser3synced through CLI and got successful access toItemZ.- Therefore,
user3managed to successfully sync with the 3 new organizations when he belonged to only one of them. - Next I added
user3toOrgX. Now he belongs toOrgXandOrgZ.user3syncs through CLI and it successfully showsItemXandItemZ. - Now,
user3logs out from CLI and logs in back again. At this point the sync fails.
The only change from step 8 to step 9 was thatuser3logged out and logged in at CLI. Nothing more.
Here’s the output from the CLI command:
bw sync Ouputs: Syncing complete.
While Nginx logs that the Sync wasn’t completed:
xxx.xxx.xxx.xxx - - [16/Oct/2020:19:00:37 +0100] "GET /api/sync?excludeDomains=true HTTP/1.1" 400 226 "-" "Bitwarden_CLI/1.12.1 (LINUX)"
And Bitwarden Service logs nothing on the failed sync attempt.
And then I run the CLI command:
bw sync --last Ouputs: 1970-01-01T00:00:00.000Z
Now, user3 got removed from OrgZ. Now he only belongs to OrgX. Syncing again from CLI ouputs the same error as above.
But if user3 logs out and in, then the sync goes through without issues and he’s able to see his own private Item and also ItemX.
This very same error also happens if user3 tries to sync while belonging to OrgY-OrgZ combination, and also to OrgX-OrgY combination.
@iamb, sorry i haven’t had time yet to check this and follow your steps. Thanks for this, i will try to do the same and hopefully i can reproduce it also.
@iamb
I have tried this now several times, but i can’t seem to reproduce the same issue as you.
I do miss some information.
When inviting the user to the org what items did you selected?
- User Type
- Access Control
- Selected a collection and “hide passwords” and “read only”
I did however found an other bug, but that doesn’t produces a sync error.
Would it be possible for you to create those test users and orgs and provide me a copy of the database so that i can test this locally?
Perfect, I’ll backup the current database to create a blank one and create some new data to reproduce the issue.
Hi again, here’s the sample DB that I just created.
removed link
I’ve created in there 3 users:
- user1
- user2
- user3
The password is the same for the 3 users: slept-refinance-overtake6
Each user has one private item in the vault, and is able to sync through cli without issues (at this point).
Then, user1 created 3 organizations: OrgA, OrgB, OrgC.
Each org contains an item.
At the moment user1 was in more than 1 org he was unable to sync through bw-cli. Syncing through web/desktop always worked fine.
User2 got into OrgA. He’s always fine syncing through bw-cli, and sees his own private item plus the item belonging at OrgA.
User3 got into OrgB and OrgC. At the point user3 belonged to more than 1 org the syncing failed through cli.
Hope that helps!
Great, I’ll see if i can test this today or tomorrow.
Thx!
@iamb
I’m sorry to say that it didn’t.
With the db you provided me i am able to, login and sync with all the users.
I have no proxy in between, just a direct connection.
It’s very very strange, i almost think it has to be either the version of bitwardenrs you are using, or something else in between. The host, the version of docker or what ever container platform you are using.
Although you have provided a lot info already, i do not see the full nginx config here.
That is the only thing left for me which could be causing something strange.
Are you maybe able to use the bitwarden cli client on the host where your bitwardenrs is running to use that to login to? And skip nginx?
I’m kinda out off ideas now, since i can’t see anything wrong with bitwarden_rs regarding this.
Also, i did had to add the diesel migrations table my self on the database you provided, it wasn’t in there.
@BlackDex Thanks for taking the time to check the DB. Now it seems clear to me that the issue might be on Nginx side.
I did setup the cli tool on the server side, and added the host as follows:
bw config server http://127.0.0.1:8000
Then login with user1 and run:
bw sync && bw sync --last && bw list items
And the sync went through without issues!
Obviously Nginx didn’t report the login & sync attempt. But the service did log the successful sync:
[2020-10-31 12:24:24.718][request][INFO] POST /api/accounts/prelogin [2020-10-31 12:24:24.746][response][INFO] POST /api/accounts/prelogin (prelogin) => 200 OK [2020-10-31 12:24:25.038][request][INFO] POST /identity/connect/token [2020-10-31 12:24:25.388][bitwarden_rs::api::identity][INFO] User user1@xxx.com logged in successfully. IP: 127.0.0.1 [2020-10-31 12:24:25.388][response][INFO] POST /identity/connect/token (login) => 200 OK [2020-10-31 12:24:25.425][request][INFO] GET /api/sync?excludeDomains=true [2020-10-31 12:24:25.473][response][INFO] GET /api/sync?<data..> (sync) => 200 OK [2020-10-31 12:24:58.316][request][INFO] GET /api/accounts/revision-date [2020-10-31 12:24:58.347][response][INFO] GET /api/accounts/revision-date (revision_date) => 200 OK
Therefore it looks like we’re on the right track.
Moving to Nginx, let me post the configuration over here:
- Bitwarden Vhost file: https://pastebin.com/3nw3iQeR
- Nginx General Config File: https://pastebin.com/XED8D0fx
Just a hunch here.
What happens if you disable gzip?
Nothing happens. I’ve tried bot setting gzip off and commenting it out, and that doesn’t change the sync issue.
From what I see, somehow issue with Nginx is connected to HTTP/1.1 sync queries.
HTTP/2.0 sync attempts get a 200 response from Nginx, whilst HTTP/1.1 sync attemps get a 400 error.
Not sure how to deal with that.
Yes, i see the issue/difference now.
This is what i have in my nginx config regarding the hub
Notice the proxy_http_version 1.1;
location /notifications/hub {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://127.0.0.1:3012;
}
Also see: https://www.nginx.com/blog/websocket-nginx/
Let me know if that solves the issue, because i just noticed that this isn’t documented on the wiki in the examples!
Hi, I was wishing so much that this trick would fix the issue! But it didn’t… 
Nevertheless, I’ll keep the /nofication/hub location block exactly as the one you posted.
Just in case, I’ve also tried adding the proxy_http_version 1.1; to the /notifications/hub/negotiate and the / location blocks.
But it didn’t solve the issue. Therefore I revert them back to their previous state as they appear in the pastebin link.
If it helps, I was thinking wether it’s possible to access some demo bitwarden_rs site, so I could try to do a CLI sync to a demo account belonging to multiple organizations and make sure the cli tool is working properly.