3,070 Commits over 730 Days - 0.18cph!
Merge: from mt_snapshot_saving
- Bugfix for ExactArrayPool not being thread-safe
Tests: TestMTSave
Bugfix: rewrite ExactArrayPool to be a ConcurrentDictionary
Was not MT safe for Jobs 4 parallel serialization as it could lazy initialize
Tests: TestMTSave
Update(tests): make SendEntityUpdates perf test spawn as many recievers as threads available
Allows to fully saturate the CPU in a test, giving a bit fairer comparison between Jobs 3 and 4
Tests: ran new test
Merge: from mt_snapshot_saving
- Optim: in UsePlayerUpdateJobs 4 entitiy saving for network is done in parallel. Use server.ParallelNetworkQueueBatchSize to tweak number of tasks
Tests: 2k procgen, built a base and a boat, rode vehicle, heli, zipline, boat, horse, fly across the world. ran all TestMTSave unit tests
Clean: fixed whitespacing
Tests: none, trivial change
Buildfix: add missing server guard
Tests: compile sim
Update: add an default TransformHandle check to TransformEx.Unsafe.ExtractTranformAccess
Better we lose a smidge of perf than have a native crash
Tests: ran all TestMTSave
Update: add GetPosMT and GetRotMT to TransformEx
- bugfix Elevator Save accessing transform
chcked internals, should be safe
Tests: fly across all of 2.5k procgen, no errors triggered
Bugfix: avoid resolving transformHandle on an optional BatteringRam.doorPhysicsHinge
Checked other cases I've modified, rest are required
Tests: none, trivial change
Bugfix: resolve game controlelr at init of BaseCardGameEntity
- use cached controller instead of trying to lazy create it in Save
Avoids invoking non MT-safe APIs in save
Tests: visited bandit camp on procgen, no errors. Ran TestMTSave tests
Bugfix: avoid NRE if bike doesn't have a sidecar
Tests: spawned a motorbike
Update(tests): TestServer now also creates a dummy BiomeMap
Not sure why pre-merge TestMTSave(GrowableEntity) didn't trip over, but sorted now
Tests: all TestMTSave
Clean: add comments for new perf test
Tests: none, trivial change
New(tests): SendEntityUpdates perf test, creates a bunch(100, 1000, 10000) of players that get sent to 10 players
Runs Jobs 3(parallelize dependency chains) vs Jobs 4(parallelize queues) serialization modes and whether to use network cache. Doesn't create any entity hierarchy (best case for Jobs 3). With network cache, Jobs 4 beats by ~2x (10k: 16ms vs 9ms), without cache Jobs 4 scales much better, up to 3x faster (10k: 62ms vs 14ms).
Tests: ran SendEntityUpdates
Update(tests): assign prefabIDs to entities created in TestServer
Avoids tests derping out trying to save global managers/players
Tests: SendEntityUpdates perf test passes
Bugfix: fix double-free on a pooled buffer
This was causing a buffer to be shared between 2 tasks when it was fetched from the pool, leading to chaos
Tests: ran SendEntityUpdates perf test (gonna submit in a bit)
Update: Jobs 4 - intercept exceptions from SendAsSnapshot/ShouldNetworkTo and log once per task
- Also if ToStream threw, don't save the resulting buffer as a network cache, as that would poison everything
Instead of bubling them all up to the top like usual, handle locally by just skipping sending the data, and report error once to avoid filling console
Tests: booted into craggy with CCTV fix reverted, no more spam
Optim: Implement parallel serialization in Jobs 4 mode
- fix CCTV_RC doing GetComponentInParent on an MT path, cache on init
- add server.ParallelNetworkQueueBatchSize, controls how many players end up being batched in a task. Higher number - less tasks
Offloading ShouldNetworkTo, serialization and sending all to worker threads. Main thread steals last task to run on main thread before spinning for other task completion.
Tests: booted craggy, gathered resources, build a base, rode ahorse, shot a gun, drove a car, heli, zipline, built a boat, used codelock
Update: rewrite ToStreamForNetwork to be thread safe
- EntityMemoryStreamPool is now a concurrent queue
Now, theoretically, we just throw it at threads and drink champagne
Tests: craggy, built a base and checked decay timer
Update: initialize every SaveInfo with ThreadSafeTime.TakeSnapshot
- also removed some jobs 2 code I missed previously
Tests: on craggy rode a horse, built a base, checked decay timer was updating as I added/removed resources
Bugfix: avoid multiple BuildingPrivilege.CalculateProtectedMinutes calls in a frame for 1 entity
Atomic latch wasn't correctly checking if it was updating from an old value only, so we could CompareExchange(1,1,1) and pass
Tests: none
Update: avoid Time calls in BasePlayer.GetUnderwearSkin
- UnderweatManifest now initialized in ServerMgr.StartServer
Tests: switched to jobs 4 and toggled underwear
Update: in UseParallelSaves mode, Ragdol component samples transform handles
Tests: on playground enabled Jobs 4 and ragdolled a bunch of times
Update: use handles to get pitch and yaw rotations for AttackHelicopterTurret in UseParallelSaves mode
- amended test to cover turret saving path
Tests: TestMTSave pass
Update: move OrientableLight interpolation invoke from Save to rpc + ondeply
Scheduling invokes won't be safe in MT
Tests: none
Update: avoid transform call in TravellingVendor.Save when it's following a spline
Tests: none
Update: move Privelege caching from DisplayingBoxStorage.Save to OnPrivelegeUpdated
We can't have it in Save as it is self-modifying and can call Vis.Entities - lets see how this holds
Tests: none
Update: force TimeZone caching in WipeTimer.ServerInit
While smashing TestMTSave(WipeTimer) I saw it NRE while resolving timezone. Couldn't repro again, but this should make it safe
Tests: none, trivial change
Update: get rid of WipeTimer rounding to second, just pass DateTime from external source
Old version would always have a sporadic fails when serialization test would run across second boundary, this way we can control it in tests
Tests: a bunch of times for hacked version of TestMTSave(WipeTimer) that has 1k serialization tasks
Update(tests): add various missing TestServer setup and unit testing bypass for entity logic
Tests: ran all TestMTSave, no sporadic failures, except WipeTimer - guess my prev fix didn't work >.>
Update: made MarketTerminal.Save MT friendly
- get rid of TimeUntil aliases from Entity.proto
- ran protobuf gen
Doesn't need protocol update, as the underlying type/serialization doesn't change
Tests: TestMTSave - all 636 tests pass, though gotta work on cleanup consistency
Update: WipeTimer now returns time rounded to a second
No change for gameplay, but makes serialization tests consistent
Tests: 20 runs of TestMTSave(WipeTimer), all pass
Update(tests): TestMTSave - move protobuf serialization to worker threads to catch more invalid API usages
Revealed 2 failures
Tests: ran all TestMTSave - MarketTerminal (TimeUnitl) and WipeTimer(missmatching stream) failed
Bugfix: avoid Time call in ChickenCoop.Save to make it MT friendly
- added chicken as part of test setup to trigger this path
Tests: TestMTSave pass
Update: BasePlayer - don't mutate own model state on save
We modify the copy as we send it, and the original value should be modified on every tick, so should be safe
Tests: none, trivial change
Update: make ItemBlueprint.FindBlueprint MT friendly
GetComponent is not thread safe, so caching item->blueprint pairs at init time
Tests: setup TC, made sure it has right upkeep calculated
Bugfix: BuildingPrivlidge(sic) - add GetOrUpdateProtectedMinutes that's MT safe and does a bit of a cheeky
If we end up sending this entity to many players in 1 frame, it will update the cached protected minutes only for 1 player, and the rest are in limbo if they got old or new value. This gives us same perf as original lazy-eval solution.
Tests: TestMTSave - got 0 fails!
Update(tests): patch MapMarkerMissionProvider
Tests: TestMTSave passes (down to 2)
Update: TestServer now initializes PaintballColorLookup.instance
Tests: TestMTSave - 5 -> 3 fails
Bugfix: make TravelingVendor.Save MT friendly
- get rid of wheel flags processing, they were updated on server and sent over, but not used
- cache linear velocity every fixed step (we're missing on interpolation, but we only send the snapshot every fixed update anyway)
- use manually interpolated steeringAngle instead of querying the wheel collider
Tests: TestMTSave(down to 5 remaining), ran TravelingVendorTest scene and looked at the vendor as it drove around
Bugfix: move tin can updates out of Save, as it's not MT safe
Tests: TestMTSave and placed it on the ground, walked over and opened inventory
Update(tests): TestServer now also initialized TrainWagonLootData and Translate classes
Tests: TestMTSave - 13 -> 7 fails
Bugfix: batch of bugfixes to Save being not MT safe
- removed more Time dependency
- replaced Transform access with TransformHandle bypass
Tests: TestMTSave - 20 -> 13 fails
Bugfix: move Signage.EnsureInitialized from Save to ServerInit, as it's not MT safe
Tests: TestMTSave - 25 -> 20 fails
Bugfix(tests): batch of patches for unit test spawned entities to avoid NREs in Save
Tests: TestMTSave - 28 -> 25 fails
Bugfix: move CarvablePumpkin.EnsureInitialized out of Save, since it's not thread safe
Tests: TestMTSave(CarvablePumpkin) and spawned and carved one on playground
Bugfix: batch of Transform replacements with TransformHandle in save code when using Jobs 4
Tests: TestMTSave - 32 -> 29 fails
Bugfix: batch of replacement for Time usage with ThreadSafeTime
Only 1 skipped out for now, BuildingPriviledge - that one does lazy-eval, which is problematic
Tests: TestMTSave - 42 -> 32
