branchrust_reboot/main/mt_snapshot_savingcancel
185 Commits over 62 Days - 0.12cph!
Bugfix: avoid MRU modification when accessing it in BasePlayer.Save
Originally intended implementation, looks like I derped awhile ago.
Tests: booted into craggy with Jobs 4
Bugfix: avoid thread race modifying same list when saving DisplayBoxStorage
- cache is now an array rather than a list
Tests: setup a display box storage, showed items in it, could see contents inside change
Bugfix: eagerly instantiate PaintballColorLookup to avoid it happening on Save, which is not mt safe
Tests: spawned on craggy, switched on jobs 4 and shot with a paintball gun
Bugfix: cache owner entity of HittableByTrains on init
Avoids lookups while running SaveComponent
Tests: spawned a wooden double barricade with jobs 4 enabled
Bugfix: make Underwear.ValidateForPlayer MT friendly on server
Fixes BasePlayer saving throwing unity exceptions
Tests: changed underwear with Jobs 4
Bugfix: make FishMount.Save mt friendly - cache fish container index
Tests: placed a fish in a trophy display with Jobs 4
Update: wrap InvokeHandler internals in DebugMTLock
Tests: booted on craggy and ran around with EnableDebugMTLock
Update: add single-thread reentrant support to DebugMTLock to avoid deadlocks
- added unit tests
Because of course InvokeHandler is non-trivially recursive
Tests: uni tests
Update: Move UnsafeScriptingAccess and DebugMTLock to Facepunch.UnityEngine
- Cleanup empty asms
Tests: ran relevant unit tests
Update: explicitly capture callstack when DebugMTLock is tripped
- frame count can be limited to via unsafe.DebugMTLockMaxFrames (def 5)
Tests: ran unit tests
New: DebugMTLock - a named lock guard that print an error on concurrent access
- controlled by Unsafe.EnableDebugMTLock, disabled by default
- codegen, unit tests
Need it to help chase down unobvious thread races, like InvokeHandler
Tests: ran unit tests
Optim: Jobs 4 - reduce server.ParallelNetworkQueueBatchSize to 16 by default
256 was waaay to high for the per-frame workload we get, it basically disabled parallelization
Tests: none, trivial change
Bugfix: fix 6 TestMTSave failures
No legit mt issues, just new entity types with missing unit test setup
Tests: ran all TestMTSave tests, pass
Bugfix: make card table saving MT-friendly
Done by removing the concept of item "busy" or "locking" - was never set but checked against Time.time
Tests: none, trivial change
Bugfix: cache rocket ammo def to avoid get_gameObject calls in AttackHelicopterRockets
Tests: spawned attack heli and geared it up - could see them in inventory
Bugfix: avoid a thread race on PlayerState MRU dictionary when saving BasePlayer
- use readonly access in Save that doesn't update MRU
- push to cache on player connect
Tests: ran CraggyIsland_missions, completed a mission - saw states change in UI
Update: avoid calling db queries when saving BasePlayer.PersistentPlayerInfo
- instead resolved on player join
It should be safe according to docs(mt calls are serialized), but this way is just simpler.
Tests: booted into craggy
Bugfix: use transform handles when saving ApartmentRoom.GetDailyUpkeepCost in Jobs 4 mode
Bypasses transform access which avoids tripping up on unity MT checks
Tests: rented a room in Jobs 4 mode, had a breakpoint in the code - no exceptions
Bugfix: cache ammo for heli turret, avoid TryGetComponent in Save which is not safe in MT
Tests: spawend a heli, setup an ak turret with 100 bullets, shot in Jobs 4 mode - no errors
Bugfix: remove duplicate transformHandle property
Prevents error in editor about duplicate serialization. Though the member is private and shouldn't serialize anyway?
Tests: tried opening & saving the rpefab, but couldn't cause original error
Bugfix: use cached time snapshot for server occlusion snapshot sending
Prevents Time usage in worker threads
Tests: none, trivial change
Bugfix: rewrite ExactArrayPool to be a ConcurrentDictionary
Was not MT safe for Jobs 4 parallel serialization as it could lazy initialize
Tests: TestMTSave
Update(tests): make SendEntityUpdates perf test spawn as many recievers as threads available
Allows to fully saturate the CPU in a test, giving a bit fairer comparison between Jobs 3 and 4
Tests: ran new test
Clean: fixed whitespacing
Tests: none, trivial change
Buildfix: add missing server guard
Tests: compile sim
Update: add an default TransformHandle check to TransformEx.Unsafe.ExtractTranformAccess
Better we lose a smidge of perf than have a native crash
Tests: ran all TestMTSave
Update: add GetPosMT and GetRotMT to TransformEx
- bugfix Elevator Save accessing transform
chcked internals, should be safe
Tests: fly across all of 2.5k procgen, no errors triggered
Bugfix: avoid resolving transformHandle on an optional BatteringRam.doorPhysicsHinge
Checked other cases I've modified, rest are required
Tests: none, trivial change
Bugfix: resolve game controlelr at init of BaseCardGameEntity
- use cached controller instead of trying to lazy create it in Save
Avoids invoking non MT-safe APIs in save
Tests: visited bandit camp on procgen, no errors. Ran TestMTSave tests
Bugfix: avoid NRE if bike doesn't have a sidecar
Tests: spawned a motorbike
Update(tests): TestServer now also creates a dummy BiomeMap
Not sure why pre-merge TestMTSave(GrowableEntity) didn't trip over, but sorted now
Tests: all TestMTSave
Clean: add comments for new perf test
Tests: none, trivial change
New(tests): SendEntityUpdates perf test, creates a bunch(100, 1000, 10000) of players that get sent to 10 players
Runs Jobs 3(parallelize dependency chains) vs Jobs 4(parallelize queues) serialization modes and whether to use network cache. Doesn't create any entity hierarchy (best case for Jobs 3). With network cache, Jobs 4 beats by ~2x (10k: 16ms vs 9ms), without cache Jobs 4 scales much better, up to 3x faster (10k: 62ms vs 14ms).
Tests: ran SendEntityUpdates
Update(tests): assign prefabIDs to entities created in TestServer
Avoids tests derping out trying to save global managers/players
Tests: SendEntityUpdates perf test passes
Bugfix: fix double-free on a pooled buffer
This was causing a buffer to be shared between 2 tasks when it was fetched from the pool, leading to chaos
Tests: ran SendEntityUpdates perf test (gonna submit in a bit)
Update: Jobs 4 - intercept exceptions from SendAsSnapshot/ShouldNetworkTo and log once per task
- Also if ToStream threw, don't save the resulting buffer as a network cache, as that would poison everything
Instead of bubling them all up to the top like usual, handle locally by just skipping sending the data, and report error once to avoid filling console
Tests: booted into craggy with CCTV fix reverted, no more spam
Optim: Implement parallel serialization in Jobs 4 mode
- fix CCTV_RC doing GetComponentInParent on an MT path, cache on init
- add server.ParallelNetworkQueueBatchSize, controls how many players end up being batched in a task. Higher number - less tasks
Offloading ShouldNetworkTo, serialization and sending all to worker threads. Main thread steals last task to run on main thread before spinning for other task completion.
Tests: booted craggy, gathered resources, build a base, rode ahorse, shot a gun, drove a car, heli, zipline, built a boat, used codelock
Update: rewrite ToStreamForNetwork to be thread safe
- EntityMemoryStreamPool is now a concurrent queue
Now, theoretically, we just throw it at threads and drink champagne
Tests: craggy, built a base and checked decay timer
Update: initialize every SaveInfo with ThreadSafeTime.TakeSnapshot
- also removed some jobs 2 code I missed previously
Tests: on craggy rode a horse, built a base, checked decay timer was updating as I added/removed resources
