Hello all! im baaaaaaaaack



I have a quick suggestion, im not sure if the dev's can incorporate this into the game with out rewriting the whole base of it, but there is a game you can look to that deals with sync issues REMARKABLY well, and that is Civ5.



every few turns the game does a sync check and whenever a client is out of sync, it "reloads" the game to bring the data back into the other clients or vice versa, causing some delay in turns, everyone in the game has to reload really quickly, but it fixes the sync issues.



this is something to look into as it is a genius way to deal with desyncs. Please implement this!

0 replies?.. >.>



Can i at least get a reply from a dev acknowledging that they're aware of this method? it's very dynamic and redundant, im no game dev, but this is the best thing i can think of.

anyone? lol

I can't speak for the devs for how this could be implemented. Obviously if it wasn't detrimental to game play in terms of game speed then it would be good, but if it would cause 10 second delay every 5 turns, then it's not worth it, imo.

It's very well do-able to check for a desync. Easy way to do it would be to just grab every important number in the game, the dust/industry/population/approval/owner/construction que of every system, the position/health/owner/stats of every ship/fleet, the research/dust/tradecontracts of every player, and then just add them all together. Maybe with a checksum or xor or something like that. It doesn't have to be accurate, as long as if something is only one bit off (which will be the case when most of the desyncs occur), the results are different.





Whilst I do have a little of gamedev experience, I'm not going to go pretend I know exactly how Endless Space works, but given that I have some experience with desyncs (and I understand the great frustration that comes from tracking those causes down), I would like to put forward the following suggestion:



At the end of every turn (As in, during the "ending turn" where nobody can do anything), make everyone do an autosave. Then compare these autosaves to the one the host has. (This comparison can be either sending over a checksum of some sort, or just the saves themselves if they aren't too big. It doesn't have to be 100% accurate, as long as it's the case that if one number is different, the result is different.)



If one of the saves is different from the host, tell the players this.





And that's it. It could be considered dirty, but it works.

Just release that, then players can play their games and know when it does/doesn't work, and rollback an autosave if necessary.





Yes, it does not fix sync issues.



But it at least tells the player when it goes wrong, allowing them to reload an autosave from the turn before that. (Which might also be useful if you're doing some testing yourselves.)



And, worst case, the "ending turn" where nobody can do anything lasts a second or two longer. Something most people will gladly sacrifice for being able to detect desyncs as soon as they happen.

good point, i've actually had that method within my freind group, when we get to turn 50 we compare some of our non vital diplomatic stats (i.e. "everyone sees orange's score at 787, right?"), and check to make sure no 0/0 fleets are amock, - sadly, in public MP's this is a little inconvenient, and people wont often rejoin to reload and fix a desync :l



but Strider, i've yet to completley playthrough a single game longer than turn 100 with out a game breaking desync, by that time the players are too burnt out to reform the game lobby, and the game goes unfinished, im sure alot of people would want this, and if its an OPTIONAL thing to add into the game on load, which woudlent' be hard to implement, its a win win.





Again to reiterate, civ 5 only causes delay/reloading of the game when there is a confirmed desync. the checks wont add delay at all, maybe a second or two at most as despiser said

Hi guys,



I've read your suggestion and it is not as simple as it may look.

We already have a desync detection mechanism for debug purpose, but it's not suitable for public release as it takes time to compute and tends to provide fault positives.

It would also take some rework to implement the correction mechanism.



But we'll keep it in mind.



Hef.

Hef wrote:
Hi guys,



I've read your suggestion and it is not as simple as it may look.

We already have a desync detection mechanism for debug purpose, but it's not suitable for public release as it takes time to compute and tends to provide fault positives.

What is your approach on the desync detection then, that makes it take so long, or provide false positives frequently?

Hef wrote:
It would also take some rework to implement the correction mechanism.

Whilst a correction mechanism would be ideal when there's a desync detection in place, it does not have to be part of the solution.



If the game itself would be able to tell when it's in sync and when it's not, than I believe that that already pulls it out of the "unplayable" status in which the multiplayer is now, since it wont leave players playing for 10 turns only to realise that all those things never really happened.

Hef wrote:
Hi guys,



I've read your suggestion and it is not as simple as it may look.

We already have a desync detection mechanism for debug purpose, but it's not suitable for public release as it takes time to compute and tends to provide fault positives.

It would also take some rework to implement the correction mechanism.



But we'll keep it in mind.



Hef.

Glad to finally see a reply on this matter, i understand that it may not be as easy as it looks, but a false positive, if corrected, will provide nothing more than essentially adresses being overwritten by values that are identical in every way to what they are overwriting, that's certainly not effecient, but from what i've seen, on the scale of bogging down the gmae, it's minimal.



Maybe im reading too much into Civ5? but whenever i see a sync correction, it happens maybe 5 times agame, and it only takes like 20-30 seconds, for the entire game and map to be reloaded from scratch, essentially, and with this game - its much less resource demanding, maps are smaller and load times should follow suit.

I must say that I don't understand the details of game development and programming, but it simply doesn't get into my head why these sync issues are such a big deal in this game. In principle, with turn-based games this should be EASIER than in real-time games!? Or am I missing something?



I am playing Civ4 with the same people that I play ES with and we never have such problems...



I am also a little disappointed at how rarely the devs respond to matters like this or just give more information... The community could help fixing it, no?

nexinex wrote:
I must say that I don't understand the details of game development and programming, but it simply doesn't get into my head why these sync issues are such a big deal in this game. In principle, with turn-based games this should be EASIER than in real-time games!?

That is entirely correct.

nexinex wrote:
Or am I missing something?

Yes, it is only correct in principle.



However, in software development, there are many, many, many ways to put something together, and when software gets anywhere near big (such as with any videogame ever), then someone will, at some point, choose a method that is not a great solution to a problem, and could make fixing such issues very difficult.

nexinex wrote:
I am playing Civ4 with the same people that I play ES with and we never have such problems...

That is probably because Civ4 had developers working on it from Civ3, which had developers working on it from Civ2, etc., which all have experience with these issues and know ways to prevent it.

nexinex wrote:
I am also a little disappointed at how rarely the devs respond to matters like this or just give more information... The community could help fixing it, no?

You think these responses are "rare"? This is a lot better than what I've seen from most devs.



I'm suprised they even responded to a suggestion as technical as this one, and I'm guessing it's because they're pushing the whole "GAMES2GETHER" thing.





But really, the desyncing could have many different causes, as there are many pitfalls, to name two big ones:



Is the game truly deterministic? Is every single variable proper initialized and is uninitialized memory never read? Are randomizers fully synced? Is it even the case that the floating point math brings the same results? Is every variable vital for the games synchronization left untouched by code not part of the deterministic game logic?



Do the inputs of the game synchronize? Does every client reroute their commands through a server in multiplayer? Are the inputs in order? Can they somehow get lost (packet drop)?





I'm just hoping they wont be afraid to turn to public to help fixing these bugs, if they're having trouble themselves.

DespiserOfFun wrote:
That is entirely correct.





Yes, it is only correct in principle.



However, in software development, there are many, many, many ways to put something together, and when software gets anywhere near big (such as with any videogame ever), then someone will, at some point, choose a method that is not a great solution to a problem, and could make fixing such issues very difficult.





That is probably because Civ4 had developers working on it from Civ3, which had developers working on it from Civ2, etc., which all have experience with these issues and know ways to prevent it.





You think these responses are "rare"? This is a lot better than what I've seen from most devs.



I'm suprised they even responded to a suggestion as technical as this one, and I'm guessing it's because they're pushing the whole "GAMES2GETHER" thing.





But really, the desyncing could have many different causes, as there are many pitfalls, to name two big ones:



Is the game truly deterministic? Is every single variable proper initialized and is uninitialized memory never read? Are randomizers fully synced? Is it even the case that the floating point math brings the same results? Is every variable vital for the games synchronization left untouched by code not part of the deterministic game logic?



Do the inputs of the game synchronize? Does every client reroute their commands through a server in multiplayer? Are the inputs in order? Can they somehow get lost (packet drop)?





I'm just hoping they wont be afraid to turn to public to help fixing these bugs, if they're having trouble themselves.

i personally pmed davea on this matter to get a response, lol



Also, the original Civnet, did not have any desync issues at all, that game was civilization 1 + a network component and i played over 5000 multiplayer matches, maybe more in my childhood and never had a desync, now im not trash talking amp - im just saying that the method used in this game is extremely ineffective, additionally RTS are actually in practice more stable and easy to sync up, since everything is calculated every second, your resources and all your clients data are stored on every other client, and they're actively updated every x MS, whereas in turn based games it tends to be compiled and then sent at the end of a turn, with the exception of certain ship movements - look at original SC, if you were to open up CE and change your minerals from 16 to 18, which is essentially what happens here, on a much smaller scale (14.006 dust income is truncated to 14.00 dust income on another client and after x turns 1 dust cause to purchase something leads to exponential and compounding desyncs) on sc, you'd just get dced from the game, flat out, while that was an anti cheat mechanism, it showed a very solid detection, and i never saw a false positive on it in my thousands of hours of play either.





also, packet loss is a huuuuge factor in desyncs, i found the latest patch remedied this to some degree, sending the data to server then client, but even still its a major factor in the syncing issue, even with a rolling 1% packet loss (which is bad, i know, but my isp sucks cock) i feel that is the reason why i desync so much more than everyone else lol



also, i'd say the randomizers don't present an issue at least as far as i can tell, the seeds are well documented for just about everything, i'm no programmer though and i cant say for certain where the issues lie, but it's certainly more than 2 or 3 issues that lead to desyncs. i'm very patient and i thank the devs for giving a measure of response to this thread, but i just wish there was something we could do to speed up this process, as the ONLY reason why i play less than once a week is because of this sync issue.



i find myself playing sins more than es, which i don't even like sins very much, while i adore es :c

Hef wrote:
Hi guys,



I've read your suggestion and it is not as simple as it may look.

We already have a desync detection mechanism for debug purpose, but it's not suitable for public release as it takes time to compute and tends to provide fault positives.

It would also take some rework to implement the correction mechanism.



But we'll keep it in mind.



Hef.

For the love of god, we aren't asking for the world, here. We are not even asking for something that guarantees that all desynchronization issues are immediately detected and fixed without us lifting a hand.



Here is something that is NOT expensive in computation or comparison time and doesn't put overdue burdens on bandwidth either: On each end of turn, the host polls all clients for a checksum calculated over a limited set of vital statistics that are likely to be affected by the loss of synchronization and, if any of the checksums differ, put up a popup saying "out of sync". Then players know that at some time between the last save they rehosted from and this point the game has lost synchronization. And they can choose to leave the game, visit the lobby, and load the game from whatever save they want to. Sure, it isn't perfect, but it cuts significantly down on wasted time for players AND allow them to file way better bug reports than "something went wrong during the last several dozen turns and now everything is wrong in the save; I'm guessing that it was X".



Which makes it much easier for the development team to get at the causes of desynchronization as well. (Well, okay, depending on the quality of the bug reports, but on average it will.)



It can be something as simple as doing a checksum over the vital FIDS statistics of systems and nothing else, since just about ANY loss of synchronization will, eventually, result in different system data (things not built, growth rates resulting in different population, things conquered...) - and hence different checksums - and it can be incrementally improved to cover more areas, or cover areas better, as time allows. And something THAT simple can even be done in way that eliminates false positives or makes them extremely rare, unlike whatever complicated mechanism you've got running in your debug version.







Take your cue from Paradox' internal development team here. They don't have a huge development team either, but they do have complicated games with lots of data being transferred regularly and issues with clients going out of sync (OOS). They learned a decade ago that it was much, much, better to throw "out of sync" messages to their players when they were pretty sure there was a problem than to let players play on unaware.



Even if it meant that players then chose to "alt+F4, reload the game, reload the save", which was the case with their earliest games due to not cleaning up nicely after leaving a session.



Because while players were - and to this day are - annoyed with the time wasted when getting an OOS alert, abandoning game, and manually rehosting, it is not anywhere near the level of annoyance that NOT notifying them before such time as the game breaks because of some data mismatch between clients or players that the clients cannot reconcile or such time as the players have to rehost for one reason or another and the game will have to be either abandoned or replayed going far back wasting HOURS of everybody's time because EVERY save since the last rehost is suspect. Really, compared to THAT, and that IS the state of Endless Space right now, throwing the occasional false positive OOS is a very small price indeed.