3,394 Commits over 792 Days - 0.18cph!
Update: add a bunch of perf counters to try to track down the spike source
- slightly amend Enqueue to help progress swap before creating a new segment if it failed
The spikes are from GC collection events, likely triggered by excessive trashing of segments(after 10 runs - 106 created, 97 trashed)
Tests: ran AllocDeallocMTShortLived(32, false) perf test
Update: reimplement pool internals on top of LPRQ
Based on paper by Raed Romanov & Nikita Koval: https://dl.acm.org/doi/epdf/10.1145/3572848.3577485
Shows same or a bit better times than FuzzyPool, but periodically has a very spiky test (from 2ms to 40ms). Also needs further tweaking and documenting
Tests: ran unit & perf tests
Update: try using CoreCLR's ConcurrentQueueSegment as a Pool implementation
Leaks a bit in busy MT perf tests, but breaks down at 32 threads (can leak up to 50% of the buffer). Not good enough on it's own
Tests: ran perf and unit tests
Update(editor): SaveViewer - add a size check when loading a save
- also emit what was the last prefab loaded (if there was one) to help track down protobuf/save without save protocol update
Tests: used to load a borked save, didn't get stuck in an infinite loop
Merge: from fix_app_turretcontrol
- No changes, just plastic history graph cleanup
Merge: from fix_app_turretcontrol
- Bugfix for frozen turret view in Rust+ app
Tests: controlled turret in Rust+ app
Bugfix: feed controlled entities transformation matrices to Rust+ camera instead of transforms
AutoTurret has a virtualized hierarchy, so the eye transform was never moved, and the camera rendered same view angles
Tests: used Rust+ app to control the turret
Merge: from codeanalyzer_getwrapperattrib
- Update Rust.CodeAnalyzer for PoolAnalyzerGetWrapperAttribute support
- Bugfix 4 pool leaks from above attrib
Tests: compiler simulator passes
Bugfix: fixup pooling errors in 4 places - 2 in WireTool, BaseAIBrain and CommunityEntity
- added a bunch of PoolAnalyzerNonCachingAttribute to BaseAIBrain calls working on proto types
Noticed 2 cases where analysis escape didn't trigger like it should've
Tests: none, trivial changes
Update: update CodeAnalyzer with support for PoolAnalyzerGetWrapperAttribute
- tag NetRead.Proto and ProtoDelta with it
Tests: editor compile shows 4 pooling errors
Merge: from fix_unitask_taskpool
- Bugfix for not-thread-safe taskpool
Tests: unit tests + booted into craggy with Jobs 4
Update: merge up to UnitTask/d239195
- Bugfix for buggy task pool returning dupes
Tests: booted into craggy, unit tests
Bugfix(editor): WhatUsesThis - don't require a whitespace after m_EditorClassIdentifier
- wrap the serach into a try-catch-throw block so we can quickly check if there's anything wrong with the prefab
Whitespace got lost in the prefab asset, causing the tool to trip up
Tests: ran it, got results
Merge: from mt_snapshot_saving
- Bugfix for duplicate field serialization
- Bugfix for unsafe Time access on worker thread
Tests: Booted into craggy
Bugfix: remove duplicate transformHandle property
Prevents error in editor about duplicate serialization. Though the member is private and shouldn't serialize anyway?
Tests: tried opening & saving the rpefab, but couldn't cause original error
Bugfix: use cached time snapshot for server occlusion snapshot sending
Prevents Time usage in worker threads
Tests: none, trivial change
Merge: from mt_snapshot_saving
- Bugfix for ExactArrayPool not being thread-safe
Tests: TestMTSave
Bugfix: rewrite ExactArrayPool to be a ConcurrentDictionary
Was not MT safe for Jobs 4 parallel serialization as it could lazy initialize
Tests: TestMTSave
Update(tests): make SendEntityUpdates perf test spawn as many recievers as threads available
Allows to fully saturate the CPU in a test, giving a bit fairer comparison between Jobs 3 and 4
Tests: ran new test
Merge: from mt_snapshot_saving
- Optim: in UsePlayerUpdateJobs 4 entitiy saving for network is done in parallel. Use server.ParallelNetworkQueueBatchSize to tweak number of tasks
Tests: 2k procgen, built a base and a boat, rode vehicle, heli, zipline, boat, horse, fly across the world. ran all TestMTSave unit tests
Clean: fixed whitespacing
Tests: none, trivial change
Buildfix: add missing server guard
Tests: compile sim
Update: add an default TransformHandle check to TransformEx.Unsafe.ExtractTranformAccess
Better we lose a smidge of perf than have a native crash
Tests: ran all TestMTSave
Update: add GetPosMT and GetRotMT to TransformEx
- bugfix Elevator Save accessing transform
chcked internals, should be safe
Tests: fly across all of 2.5k procgen, no errors triggered
Bugfix: avoid resolving transformHandle on an optional BatteringRam.doorPhysicsHinge
Checked other cases I've modified, rest are required
Tests: none, trivial change
Bugfix: resolve game controlelr at init of BaseCardGameEntity
- use cached controller instead of trying to lazy create it in Save
Avoids invoking non MT-safe APIs in save
Tests: visited bandit camp on procgen, no errors. Ran TestMTSave tests
Bugfix: avoid NRE if bike doesn't have a sidecar
Tests: spawned a motorbike
Update(tests): TestServer now also creates a dummy BiomeMap
Not sure why pre-merge TestMTSave(GrowableEntity) didn't trip over, but sorted now
Tests: all TestMTSave
Clean: add comments for new perf test
Tests: none, trivial change
New(tests): SendEntityUpdates perf test, creates a bunch(100, 1000, 10000) of players that get sent to 10 players
Runs Jobs 3(parallelize dependency chains) vs Jobs 4(parallelize queues) serialization modes and whether to use network cache. Doesn't create any entity hierarchy (best case for Jobs 3). With network cache, Jobs 4 beats by ~2x (10k: 16ms vs 9ms), without cache Jobs 4 scales much better, up to 3x faster (10k: 62ms vs 14ms).
Tests: ran SendEntityUpdates
Update(tests): assign prefabIDs to entities created in TestServer
Avoids tests derping out trying to save global managers/players
Tests: SendEntityUpdates perf test passes
Bugfix: fix double-free on a pooled buffer
This was causing a buffer to be shared between 2 tasks when it was fetched from the pool, leading to chaos
Tests: ran SendEntityUpdates perf test (gonna submit in a bit)
Update: Jobs 4 - intercept exceptions from SendAsSnapshot/ShouldNetworkTo and log once per task
- Also if ToStream threw, don't save the resulting buffer as a network cache, as that would poison everything
Instead of bubling them all up to the top like usual, handle locally by just skipping sending the data, and report error once to avoid filling console
Tests: booted into craggy with CCTV fix reverted, no more spam
Optim: Implement parallel serialization in Jobs 4 mode
- fix CCTV_RC doing GetComponentInParent on an MT path, cache on init
- add server.ParallelNetworkQueueBatchSize, controls how many players end up being batched in a task. Higher number - less tasks
Offloading ShouldNetworkTo, serialization and sending all to worker threads. Main thread steals last task to run on main thread before spinning for other task completion.
Tests: booted craggy, gathered resources, build a base, rode ahorse, shot a gun, drove a car, heli, zipline, built a boat, used codelock
Update: rewrite ToStreamForNetwork to be thread safe
- EntityMemoryStreamPool is now a concurrent queue
Now, theoretically, we just throw it at threads and drink champagne
Tests: craggy, built a base and checked decay timer
Update: initialize every SaveInfo with ThreadSafeTime.TakeSnapshot
- also removed some jobs 2 code I missed previously
Tests: on craggy rode a horse, built a base, checked decay timer was updating as I added/removed resources
Bugfix: avoid multiple BuildingPrivilege.CalculateProtectedMinutes calls in a frame for 1 entity
Atomic latch wasn't correctly checking if it was updating from an old value only, so we could CompareExchange(1,1,1) and pass
Tests: none
Update: avoid Time calls in BasePlayer.GetUnderwearSkin
- UnderweatManifest now initialized in ServerMgr.StartServer
Tests: switched to jobs 4 and toggled underwear
Update: in UseParallelSaves mode, Ragdol component samples transform handles
Tests: on playground enabled Jobs 4 and ragdolled a bunch of times
Update: use handles to get pitch and yaw rotations for AttackHelicopterTurret in UseParallelSaves mode
- amended test to cover turret saving path
Tests: TestMTSave pass
Update: move OrientableLight interpolation invoke from Save to rpc + ondeply
Scheduling invokes won't be safe in MT
Tests: none
Update: avoid transform call in TravellingVendor.Save when it's following a spline
Tests: none
Update: move Privelege caching from DisplayingBoxStorage.Save to OnPrivelegeUpdated
We can't have it in Save as it is self-modifying and can call Vis.Entities - lets see how this holds
Tests: none
Update: force TimeZone caching in WipeTimer.ServerInit
While smashing TestMTSave(WipeTimer) I saw it NRE while resolving timezone. Couldn't repro again, but this should make it safe
Tests: none, trivial change
Update: get rid of WipeTimer rounding to second, just pass DateTime from external source
Old version would always have a sporadic fails when serialization test would run across second boundary, this way we can control it in tests
Tests: a bunch of times for hacked version of TestMTSave(WipeTimer) that has 1k serialization tasks
Update(tests): add various missing TestServer setup and unit testing bypass for entity logic
Tests: ran all TestMTSave, no sporadic failures, except WipeTimer - guess my prev fix didn't work >.>
