Currently, you cannot use custom factions in multiplayer this is an issue I've seen other people on the steam forums complaining about. Even when the "allow custom race" option is ticked on, it will not allow you to play a custom race. You are given the option to pick the custom race, but when you click on them it will briefly show your custom race then revert back to showing the last original race you had selected.  If you had the race selected before you invite someone to the game lobby it will work fine until you select a different faction, then the problem mentioned above will occur again.

been five years and and still has not been fixed