1,947 Commits over 488 Days - 0.17cph!
Buildfix: add missing namespace using
Tests: none, trivial change
Merge: from analytics_track_menu_times
- Client analytics now aggregates frametimes, lag spikes and gc events into InGame, MainMenu, Inventory, Crafting, Contacts and Map groups
- Skip sending lag spikes and gc counts if there were none during performance logging tick
Tests: observerd client_perf analytics ticks in editor with debug code
Clean: remove unused namespace using
Tests: none, trivial change
Clean: remove debug logging
Tests: compiles in editor
Optim: don't emmit lag spikes and gc counts if there weren't any during analytics ticks
Tests: observed couple client_perf ticks in editor
Bugfix: no more use-after-free for Frametimes
- also reimplemented the fix from previous CL, as it was part of the problem
Tests: in editor spawned on craggy and opened main menu for multiple client analytics ticks - no more exceptions
Update: Client Analytics - categorize client frametimes, gc_count and lagspikes
- Bugfixing leaking a couple List<TimeSpan> containers
We now filter those out to InGame, MainMenu, Inventory, Crafting, Contacts and Map frame types and aggregate per type. Everything apart from InGame aggregates slightly reduced info set. New analytics is backwards compatible.
Tests: left it running with debug logging. relevant info seems to come through. There's a bug with use-after-free of List<TimeSpan> - will fix next
Update: convert error logs to error-once logs
The current changes mitigate the problem, which will allow peeps to play longer while we chase up what's going wrong.
Tests: none, trivial change
Update: adding more logging in an attempt to catch baseplayer NREs
Tests: 2p local session with spectating, disconnects
Clean: leaving a TODO note for one I return to this
Update: if either logging or dry run is enabled, log uploader activity
Previously it was editor only, and you had to enable dry-run specifically to get those logs. This makes it easier to work with.
Tests: none, trivial change
Bugfix: in editor with SERVER+CLIENT send client records to client gameplay AnalyticsTable
Not sure if it's technicall bug, but it did differ from original logic
Tests: logged activity in the editor
Optim: wind down AzureAnalyticsUploader when UseV2 is active
Gets rid of async tasks that are generating allocs while waiting for work
Tests: in editor activated v2, waited till upload happens, then disabled and waited for normal upload
Update: name the analytics thread
Tests: dry run in the editor
Optim: reduce allocations by relying less on async/await
- Use a Thread instead of a Task for internal AnalyticsManager work (this also fixed editor 10s shutdown delay)
- Allow internal upload tasks to run in parallel instead of waiting on sequential completion
Previously it would frequently GC, now it's back to normal
Tests: ran in editor in dry run mode. ran unit tests
Update: add profiling markers to AnalyticsManager and it's uploaders
Tests: ran in editor
Update: FPUploaderImpl - add a warning in case someone attempts to feed it non-json AnalyticsTable
Tests: none, trivial change
Update: append AnalyticsTable name when logging accumulator activity
Tests: none, trivial change
Update: accumulator log now also outputs byte size (pre compression if there is one)
Tests: none, trivial change
Bugfix: fix false "EventRecord pooling error" when disabling analytics.usev2
Hoping to rip it out once we fully convert to v2
Tests: unit tests + enabled-disabled in editor
Update: move analytics-for-server shutdown to ServerMgr.CloseConnection
This shuts down server before steamwroks in SERVER+CLIENT editor - makes editor's lifecycle management less confusing
Tests: confirmed client session end event still fires in editor
Bugfix: ensure session end event gets enqueued for V2 analytics
Tests: monitored in editor, saw it when stopping player
Clean: move new implementation into it's own file
Might split it up further, but not needed for now
Tests: editor compiles
Bugfix: respect rate limiters of AnalyticsTables and avoid mixing data when uploading
- Fixed by creating an accumulator per AnalyticsTable that's managed by uploaders
I need to profile and work on allocs, but it seems to be working
Tests: unit tests and monitored client telemetry
Update: Integrate the new AnalyticsManager into existing logic
- Guarded by `analytics.usev2` (default off for now)
- Bugfix for rate-limiting being disabled due to mixing Now and UtcNow (whoops)
- Added global `analytics.dryrun` and `analytics.log`
Did a cursory test in editor using dry running and it seems to be working, but it leaks performance data into player_tick blobs. Need to reorganize accumulators again - always-aggregate isn't always valid.
Tests: used above switches and monitored data flow.
Update: Add FallbackUploader - uses primary uploader if enabled, otherwise uses fallback
- Added a test that checks it
- DryRun is now a per-accumulator settings instead of a global overwrite - lets each uplaoder decide how to act
Allows us to setup flexible chains(extra loggers, backups, etc), and more importantly we can emulate current analytics flow
Tests: ran unit tests
Update: Move accumulators into uploaders
- AnalyticsTests.TestUploade has logging enabled permanently to help validate things
It was a blocker to implement uploader chains (which I discovered we need for server, but should be quick)
Tests: ran unit tests
Clean: rename AnalyticsUploader -> AnalyticsManager
Disambiguates from the internal IUploader
Tests: compiles in editor
Bugfix: apply separator in json-lists aggregation correctly
Tests: used accumulator logging to see what is being fed
Update: Accumulators can now log what they're being fed
This revealed a bug with json concatenation, should be a quick fix
Tests: ran unit tests
Merge: from server_occlusion_gen_reorder_narrowphase
- Bugfix for ServerOcclusion potentially crashing during generation
Tests: generated custom map that had a consistent crash and a default 4.5k procgen editor world
Bugfix: ServerOcclusion - don't schedule same SubGrid cells repeatedly for rock intersection
It caused more work to be done(this saves ~13s), and in some cases even an out-of-bounds access crash.
Tests: generated server occlusion cache for a 6k custom map that had a consistent crash, generated default editor 4.5k proc gen world
Update: AnalyticsUploader - combine AzureBulkJsonUploader and AzureBulkCSVUploader into BulkAccumulator
- this also allows to combine the 2 uploaders into 1
This enabled aligning EventRecord serialization with current system. I think this is the last change to the code, next up will updating user code and testing in "real world".
Tests: ran unit tests
Update: AnalyticsTable - bring back useJsonDataObject support
Tests: ran unit tests
Tests: add all Analytics Uploader tests
- Covers ClientFPUploader, ServerFPUploader, AzureBulkJsonUploader, AzureBulkCSVUploader with multithread data generation
- Specialized editor logging for each uploader, as it doesn't make sense to emit compressed goup
ClientFPUploader will need internet connection as it relies on steam
Tests: ran unit tests
Update: AnalyticsUploader - refactor how we accumulate data
- AnalyticsTable no longer serializes EventRecords, that's moved to IAccumulators
- Implement serializers on top of IAccumulators (Csv, Json, CompressedCsv, CompressedJson)
- Uploaders automatically create IAccumulators per AnalyticsTable
Still missing an accumulator that would replicate how client analytics are serialized, but now it should be easy to do. Unit tests got simplified, need to expand coverage
Tests: ran unit tests
Update: implement AzureBulkUploaderImpl
- uses UploadBlobAsync instead of OpenWriteAsync
Not tested. Realized that current abstractions will not play well together, so gotta move a bit of stuff around.
Tests: none, will do later once I start converting user-code
Update: AnalyticsUploader - implement FPUploaderImpl
- add editor-only DryRun switch to log everything instead of actually sending it
- IUploader.Upload is now async
Tests: none, not hooked up to anything yet, that'll be next experiment
Update: AnalyticsTable - return EventRecord back to pool
- amended test to check for pool usage
Tests: ran unit tests
Update: AnalyticsUploader now flushes it's queue on shutdown.
- Updated tests to validate everything got flushed
- Fixed missed gzipstream end block serialization tracking
- Fixed ever-growing memory stream
Next up need to add EventRecord pooling, then gotta implement remaining uploaders, and in theory it's done
Tests: ran unit tests
Update: add per-AnalyticsTable telemetry
- using this telemetry to improve unit tests
Next up need to add support for flush-on-shutdown
Tests: ran unit tests
Update: Initial rewrite of the analytics core
- Only implements NullUploader, I'll implement actual uploaders after tests are good
- Added basic tests for it, but they don't validate anything - that'll be next.
- Renamed existing AnalyticsTests.cs to Test.EventRecord.cs (+ internal rename) - those tests validate EventRecord serialization
Goal is to use one implementation to cover needs of both Client and Server. Got the basics done, need to add telemetry to be able to test better, support EventRecord pooling and implement the existing uploaders.
Tests: ran new unit tests
Merge: from baseplayer_vis_nre
- Potential(low chance) bugfix for NRE on spectate end
- Breadcrumbs in case it's not fixed
Tests: editor and standalone build spectating loop
Clean: remove hacky testing code changes
Tests: editor compiles
Bugfix: potentially fix BasePlayer Vis NRE
- Added extra logs to catch other sources
- Refactored code to extract invariant checks
Not confident it'll actually solve it, if it does - can remove extra logs
Tests: using spec_loop command to force client to quickly spectate-then-stop in a loop. Tried both in editor and standalone builds - no issues(though I couldn't repro it originally this way anyway)
Update: don't build UI bundle for server bundles
Tests: none, trivial change
Clean: get rid of StoreTakeover internal warmup count
It never goes past 1 since we fixed open order of store
Tests: stuck an assert and validated poked at different menus
