jan hloušek, keen software house
TRANSCRIPT
![Page 1: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/1.jpg)
Multiplayer challenge
Jan HloušekPetr Minařík
Sandra Lenardová
![Page 2: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/2.jpg)
Keen Software House
• Founded 2010 by Marek Rosa• 50+ developers• Privately held• Games
- Miner Wars- Space Engineers- Medieval Engineers
• PC Steam (for now)
![Page 3: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/3.jpg)
Space Engineers
• 3+ years on Steam Early Access• 2M+ players• 200k+ mods on Steam Workshop• VRage engine
- C# .net 4.6- DX11- Havok
![Page 4: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/4.jpg)
Space Engineers
• Sandbox• Infinite playground• Editable / destructible environment• Voxels / Planets• Survival / Creative
![Page 5: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/5.jpg)
![Page 6: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/6.jpg)
![Page 7: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/7.jpg)
![Page 8: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/8.jpg)
![Page 9: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/9.jpg)
![Page 10: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/10.jpg)
Challenge
Internal challenge for delivering new multiplayer code• 2 teams joined competition• 3 months deadline• 1 million CZK reward• Released in SE beta on Christmas• Overwhelming community reaction
![Page 11: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/11.jpg)
• Play testing• Integrating debug tools• Experimenting
- Hacky implementations of various ideas- Play testing / Honest feedback- Wrong paths abandoned without mercy
• Coding- Removing code- The least time spent on coding actually- Removed more code than added
Development cycle
![Page 12: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/12.jpg)
Debug tools - Statistics
![Page 13: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/13.jpg)
Debug tools - Profiler
• Server / Client• Observe remote’s profiling data• Save / load for later analysis
![Page 14: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/14.jpg)
Debug tools - Profiler
![Page 15: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/15.jpg)
Debug tools - Debug screens
![Page 16: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/16.jpg)
Debug tools - Clumsy
• https://jagt.github.io/clumsy/
![Page 17: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/17.jpg)
Debug tools - Physics Remote Debugging
![Page 18: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/18.jpg)
Debug tools - Spectator
![Page 19: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/19.jpg)
• https://sourceforge.net/projects/tracetool/
Debug tools - TraceTool
![Page 20: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/20.jpg)
What we learned
• Clients were telling server what to do- Resulting in all sorts of desyncs- Ultimately resulting in physics instability and explosions
• No lag compensation- Resulting in rubber banding and jittering- Poor gameplay experience
• Time synchronization using altering physics velocities- Resulting in all sorts of desyncs- Increasing code complexity
• Universe synchronized using linear complexity- Server performance problems
![Page 21: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/21.jpg)
• Authoritative server- Client sends player inputs- Server sends the world state changes
• Physics-based prediction- To combat network lag
• Synchronized simulation steps• Spatial optimisations
- Cluster based synchronization
• Admin tools
What we changed
![Page 22: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/22.jpg)
Prediction - original state
Client 1 Client 2Server
![Page 23: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/23.jpg)
Prediction - original state with lag
Client 1 Client 2Server
Lag
![Page 24: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/24.jpg)
Prediction - removing one frame lag
Client 1 Client 2Server
Lag
![Page 25: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/25.jpg)
Prediction - corrections
Client 1 Client 2Server
Lag
corr
ection
![Page 26: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/26.jpg)
Prediction - synchronized simulation steps
Client 1 Client 2Server
corr
ection
1
2
3
4
5
6
7
8
9
1
2
3
4
5
![Page 27: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/27.jpg)
Prediction - synchronized simulation steps
Client 1 Client 2Server
corr
ection
1
2
3
4
5
6
7
8
9
1
2
3
4
5
![Page 28: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/28.jpg)
Network protocols - TCP
• Connection based
• Guaranteed reliable and ordered
• Automatic packet splitting
• Flow control
• Easy to use
![Page 29: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/29.jpg)
Network protocols - UDP
• Connection managed by ourselves
• Possibly unreliable, unordered, duplicated
• Packet size matters
• Possible line congestion
![Page 30: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/30.jpg)
• Requirements- Get the packets as fast as possible to end point- Old packets are no use for us- Dropped packets will soon be compensated with new data- Out of order packets are still an issue - lets drop them for now
• UDP is a better match
Network protocls - conclusion
![Page 31: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/31.jpg)
Prediction - synchronized simulation steps
Client 1 Client 2Server
corr
ection
1
2
3
4
5
6
7
8
9
1
2
3
4
5
![Page 32: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/32.jpg)
Prediction - server buffering / final state
Client 1 Client 2Server
corr
ection
Playout delay buffer
1
2
3
4
5
6
7
8
9
1
2
3
4
5
![Page 33: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/33.jpg)
Spatial optimizations
• Synchronize with clients their points of interest- Saving network bandwidth and performance
• Simulate only points of interest- Rest of the world is “sleeping”
• Delta time update
- Waking up entities getting into point of interest
- Simulating them in one update using the delta time while they were sleeping
- Potential problems
![Page 34: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/34.jpg)
Spatial optimisations
![Page 35: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/35.jpg)
Exceptions
• Antenna systems- Spanning large distances- Have to be simulated using custom system
• Remote controls- Point of interest is switched to the new location
• Remote cameras- Multiple points of interests for each client
![Page 36: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/36.jpg)
Admin tools
• Trash removal• Block limits• Player levels
![Page 37: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/37.jpg)
What next?
Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing
![Page 38: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/38.jpg)
And then?
Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing
![Page 39: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/39.jpg)
And after that?
Performance balancing• Master server monitoring slave servers’ load• Responsible for cluster management
- Adding- Removing- Moving
• Allowing MMO scale of player numbers
![Page 40: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/40.jpg)
Continuation of...
![Page 41: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/41.jpg)
References
• Source multiplayer guide- https://developer.valvesoftware.com/wiki/Source_Multiplayer_N
etworking
• Gaffer on games- http://gafferongames.com
• Multiplayer Game Programming- Joshua Glazer an Sanjay Madhav, Addison-Wesley Professional
![Page 42: Jan Hloušek, Keen Software House](https://reader036.vdocuments.site/reader036/viewer/2022062223/58e4f3c11a28abdf698b478f/html5/thumbnails/42.jpg)
Looking for a challenge?
• MMO scale multiplayer• SE XBOX port• Micro voxels• Vulkan rendering• Multiverse
Join our team!Contact:[email protected] 485511