My friend is playing on the Game Pass and I'm playing in Steam which means I cannot send him an invite but even when I send him my lobby ID, he still cannot join my game.

We have a workaround for now : I create a public game, he can see it in the server browser and join it and then I switch it to private.