1,882 Commits over 457 Days - 0.17cph!
Update: minor ServerProfiler function filtering adjustment
- built from b90a83e5
Was annotating everything from SubGrid, instead of ignoring everything
Tests: unit tests
Merge: from baseplayer_serverupdateparallel
- Optimizes ServerOcclusion by reordering work and reducing number of pairs considered
- Optimizes parallel snapshot send out by reducing Pool contention
- General optims via reduced profiler instrumentation (new ServerProfiler.Core binaries)
Tests: 2p session on craggy with teleporting around and various occlusion scenarios (invis, dead, normal occluded, in view)
Update: new ServerProfiler with more filters
- Built on 276b03cf
Excludes a bunch of BasePlayer, BaseEntity, BaseNetworkable light functions, Native calls to RakNet and EAC, StableCache iterators.
Tests: booted in editor, took a snapshot of 10 frames
Optim: Set Pool<BufferStream> capacity to 32k (sum of NetRead and NetWrite caps)
We end up eating up default capacity on busy servers(1.1k BufferStreams on 330p server), causing allocs. This should help avoid that.
Tests: none, trivial change
Optim: UsePlayerTasks - pre-allocate NetWrites when sending snapshot queues
This should reduce contention on the Pool<NetWrite>'s lock and let worker threads blast as fast as possible. Will need to revisit after measuring the pre-alloc overhead for main thread.
Tests: On Craggy with UsePlayerUpdateJobs 2 flew away from island's network range then back - entities were where there have been before flying away.
Merge: from spectating_disconnect_nre_fix
- Bugfix for server NRE when spectating player disconnects
Tests: 2p on Craggy, disconnected while spectating - no more NRE
Bugfix: stop spectating when disconnecting
This cleans up cached spectators on spectated player's end, which avoids server NREs and leaks
Tests: 2p on craggy, started spectating then disconnected - no NRE
Merge: from main
- brings bugfix for player not seeing items on cargo ship(and other global networked entities)
- new analytics mode (switched off)
Tests: editor compiles, spawned cargoship on craggy and threw a rock on it - it replicated
Merge: from basepalyer_netgroup_interit_fix
- Fixes cargoships(and other global entities) putting players into global network group which stops replication with "outside" world
Tests: 2p session on Craggy, spawned a cargo ship and threw items, killed other player, tried spectating
Bugfix: stop player from inheriting parent's network group
Unintentional change as part of baseplayer_vis_nre merge. This caused player to inherit global network group on cargoship, which stops replicating anything outside of it.
Tests: 2p session on Craggy, spawned a cargo ship and threw items, killed other player, tried spectating
Merge: from analytics_queue_rewrite
- Bugfix for missing session-start event from Client, invalid session-end event from Client, malformed json aggregation
- Added support to edit upload intervals (tied to existing same server vars)
- Adjusted uploaders for some tables to avoid trying to send to our backend
Tests: unit tests + manual testing on Craggy (S+C mode) and Bootstrap (C only)
▊▊▉▇▋▇: ▅▅▅▆▅█▊▌▉▊▉ - ▆▉▍▅▄▋ ▍█▆▇ ▆▉▉▋▄▊▇ ▆▊▉█ ▍▌▇▍▄▊▅▍▊▆▄▆▌▉▉▆▄ ▉▍▊▍
▉▄▌▊ ▊▉▅▌▇▊ ▌▄▅ ▇▋▉▆▄▅-▋▉▊▅▆▆▊▅▉ █▆▊▋▍▉▇▋▉ ▆▇ ▄▍-▅▇▊ ▅▅▋ ▆▋▊▌█▌ ▌▇▌▍▉▇▍ ▇▊▇ ▄▍▄▅ ▊▍▍▄▍▊▄▋ ▇▇ █▄▅▌▍▋▌
█▆▌▄▅: ▉▌▍▍▇▋▇ ▋▆▋▄ ▄▅▇▌▉▌, ▋▌▄▄▉██▆▄ █▍▉▊ ▊▆▄▉█▄▄ ▄▇▆ █▉▄▋▄▄▋▊/▍▆▉█▄▍▄▍██▋
Update(editor): AnalyticsV2 - emit session start event
This is to make validating analytics on the backend easier (as this helps to form a full session)
Tests: with logging enabled observed session-start being fired before player_connect for server is sent
Update: AnalyticsV2 - add support for changing upload intervals
- made RuntimeProfiler and TickLogger adjust their intervals
Tests: enabled runtime profiler and changed interval - confirmed sent when expected
Update: AnalyticsV2 - restore gameplay_events upload interval
Accidentally submitted while testing it yesterday
Tests: none, trivial change
Bugfix: AnalyticsV2 - don't spam "appended 0 bytes" when logging is enabled
Tests: none, trivial change
Bugfix: AnalyticsV2 - ensure events sent at shutdown have valid steam tickets
Done by pumping steam event queue while analytics thread is shutting down
Tests: unit tests and ran Client mode in editor
Bugfix: AnalyticsV2 - only recycle auth tickets if not consumed them
- Added extra checks for early steam shutdowns
Previously we would eagerly cancel tickets, which is not api-appropriate. Now only doing so if either an exception occured or we're in dry-run mode
Tests: ran unit tests. Still can't catch why auth ticket is sometimes null on shutdown
Bugfix: AnalyticsV2 - fix invalid Json aggregation
- logs are aggregated again to reduce spam
This breaks azure bulk aggregation, but now the code structure is in place to support both
Tests: ran unit tests
Bugfix: AnalyticsV2 - don't miss client-only initialization during bootstrap flow
Tests: monitored client bootstrap analytics
Bugfix: AnalyticsV2 - skip client analytics if steam hasn't been started
Tests: none, trivial change
Bugfix: AnalyticsV2 - Log and DryRun no longer skip accumulators and uploaders once changed
As an extra, it got rid of a bunch of inheritance code
Tests: ran unit tests and logged analytics in editor
Update: UsePlayerTasks - ensure SendEntitySnapshotsWithChildren_AsyncState have equally sized batches
Previously we counter players for batch limit (skipping it's hierarchy), but that could lead to lopsided unequal tasks
Tests: couldn't test this one, but it's similar code to other places so should be gud
Optim: UseOcclusionV2 - ServerOcclusion caches unoccluded connections
- removed couple redundant if checks
This allows us to completelly skip faster-but-still-slow ShouldNetworkTo that have internal hash lookups in SendNetworkPositions - hoping it'll speed it up 50%
Tests: ran in and out of occlusion on Craggy. disconnected to validate sleeper, then killed sleeper and reconnected - all works
Merge: from analytics_save
- Adds a "save" event tracking when server creates a save
Tests: triggered save in editor with logging
Update: add analytics save event
Tests: ran an explicit save in editor
Update: add extra TODO for UseOcclusionV2
Tests: none, trivial change
Update: UseOcclusionV2 - propagate networkTime to OcclusionGetRecentlySeen
Should be a smidge faster, but it's primarily to enable threading if we go that way. Also saves on profiler inhibiting the call (need to exclude it on the profiler side)
Tests: none, trivial change
Clean: restructure UseOcclusionV2 code to make profiling easier
Tests: compiles in editor
Clean: fix ShouldSkipServerOcclusSion typo
English is hard
Tests: editor compiles
Optim: UseOcclusionV2 - OcclusionFrameCache uses NetIDs instead of BasePlayer pairs
This avoids ObjectEqualityComparer which has overhead because of unity-null checks
Tests: none, trivial change
Optim: UseOcclusionV2 - reduce lastPlayerVisibility hash lookups
Should help OcclusionGatherLostPairsToSend internal loop to stay hot. Also gets rid of small funcs that are inhibited by profiler
Tests: ran in and out of occlusion
Clean: add a comment explaining the extra OcclusionUpdateLostVisibility
Tests: none, trivial change
Optim: UseOcclusionV2 - invert how we process occludees
- instead of checking if we need to send player to occludees, we process occludees against player
This caused us to miss sleepers previously, which we corrected with extra checks by reversing pairs(and potentially generating more netowrk messages). But we don't need to do all of that extra work if we do it in the right order. Hoping this cuts downstream queue 2x.
Tests: 2p on craggy - checked occlusion works, going invis works, spectating works, sleepers replicate
Clean: introduce BasePlayer.UseOcclusionV2 use it as a switch instead of UsePlayerTasks (works same way)
Makes it easier to track what's related to what
Tests: editor compiles
Optim: ServerOcclusion - use network IDs instead of BasePlayer for dictionary key
Helps avoid unity-null check overhead when looking up recently seen players
Tests: 2p session on craggy - disconnected as second and killed sleeper
Bugfix: ServerOcclusion - remove destroyed players from lastPlayerVisibility cache
We were leaking baseplayers and it was slowing down hash lookups
Tests: 2p session on craggy, 2nd player disconnected, killed their sleeper - lastPlayerVisibility was empty for the remaining player
Update: UsePlayerTasks - lifting skipping logic out of OcclusionLineOfSight to caller
- left a couple more TODOs as I found a weirdness in the original code
Only thing left in OcclusionLineOfSight is grid-cell caching and deduplication.
Tests: none, discovered unit tests are broken since I upgraded occlusion cache version a month ago. Will fix and test later
Optim: UsePlayerTasks - ServerOcclusion uses CustomShouldNetworkTo and CustomShouldSkipServerOcclusion
Microoptim - this reduces hashset look ups and moves some checks to be run earlier. Main benefit is lifting more code out of BasePlayer.OcclusionLineOfSight
Tests: none, skipping tests for now
Update: refactor ServerUpdateOcclusionParallel to allow for experimental changes
Tests: compiles in editor
Update: add a bunch of optimization TODO notes for UsePlayerUpdateJobs 2
Gathered from inspecting 330p server snapshot from a late-in-the-patch-cycle server. Will clean-up before merge.
Tests: none, trivial merge
Buildfix: add missing CLIENT guards
Tests: build all configs locally
Merge: from analytics_queue_rewrite
- New analytics uploading backend, disabled by default (analytics.usev2)
- New debug vars - analytics.log and analytics.dryrun
Tests: unit tests, used both old and new analytics backends and monitored logs
Merge: from baseplayer_vis_nre
- Simplified how debug.invis works
- Reimplemented how global.spectate and spectateid work - spectator no longer parents to the spectated player
- Tightened existing debug logs around mystery baseplayer NRE.
This eliminates a source of potential NREs related to players spectating others, but doesn't outright forbid them
Tests: 2p sesions on Craggy, one player spectates other - killed scientist with AK(hs audio replicated), helicopter fly around, motorbike ride around, noclipped outside of usual network range and back, cycled invis, left the server and returned - no issues. Tried debug camera.
▉▋▆▅▆▇: ▊▊▇▌▄▍▋ ▊▇▄▍▋▇▄▄▆▇ ▍▊▍ ▍▅▄▍ ▍▄▇▌▇▌▇▋█▄▉ ▍▇▆▋ ▍▋▅█▋▋▋▇▊▊▆ █▇ █▇▇▉▊ ▇▍▄-▉▌▇█▊ ▅▆▆█▅▌▄
▅▋▅▋▊: ▆▅▆▊▊▆▅▇▄ ▇▅▋▍▇▍ ▇▊▇▍▌ ▍▅▊█ ▅▅▉▇ █▆▋▅▉▍▇▆, ▋▇▍▉▆ ▅▊▌▄, ▉▉▋▍▋██, ▆▍▊▅█▋▉▉▊▋, ▍▌▉▄▅▄▍▋ - ▊▇ ▇▇▊▌█ ▌█▉▊▌█▊▋ ▆▌ ▊▆▆ ▋▍▌█▍█▅▌▆'▍ ██▋, ▅▅ ▅▄▌▆ ▋▊ ▊▆▍▌
