branchrust_reboot/main/analytics_queue_rewritecancel
44 Commits over 30 Days - 0.06cph!
▊▊▉▇▋▇: ▅▅▅▆▅█▊▌▉▊▉ - ▆▉▍▅▄▋ ▍█▆▇ ▆▉▉▋▄▊▇ ▆▊▉█ ▍▌▇▍▄▊▅▍▊▆▄▆▌▉▉▆▄ ▉▍▊▍
▉▄▌▊ ▊▉▅▌▇▊ ▌▄▅ ▇▋▉▆▄▅-▋▉▊▅▆▆▊▅▉ █▆▊▋▍▉▇▋▉ ▆▇ ▄▍-▅▇▊ ▅▅▋ ▆▋▊▌█▌ ▌▇▌▍▉▇▍ ▇▊▇ ▄▍▄▅ ▊▍▍▄▍▊▄▋ ▇▇ █▄▅▌▍▋▌
█▆▌▄▅: ▉▌▍▍▇▋▇ ▋▆▋▄ ▄▅▇▌▉▌, ▋▌▄▄▉██▆▄ █▍▉▊ ▊▆▄▉█▄▄ ▄▇▆ █▉▄▋▄▄▋▊/▍▆▉█▄▍▄▍██▋
Update(editor): AnalyticsV2 - emit session start event
This is to make validating analytics on the backend easier (as this helps to form a full session)
Tests: with logging enabled observed session-start being fired before player_connect for server is sent
Update: AnalyticsV2 - add support for changing upload intervals
- made RuntimeProfiler and TickLogger adjust their intervals
Tests: enabled runtime profiler and changed interval - confirmed sent when expected
Update: AnalyticsV2 - restore gameplay_events upload interval
Accidentally submitted while testing it yesterday
Tests: none, trivial change
Bugfix: AnalyticsV2 - don't spam "appended 0 bytes" when logging is enabled
Tests: none, trivial change
Bugfix: AnalyticsV2 - ensure events sent at shutdown have valid steam tickets
Done by pumping steam event queue while analytics thread is shutting down
Tests: unit tests and ran Client mode in editor
Bugfix: AnalyticsV2 - only recycle auth tickets if not consumed them
- Added extra checks for early steam shutdowns
Previously we would eagerly cancel tickets, which is not api-appropriate. Now only doing so if either an exception occured or we're in dry-run mode
Tests: ran unit tests. Still can't catch why auth ticket is sometimes null on shutdown
Bugfix: AnalyticsV2 - fix invalid Json aggregation
- logs are aggregated again to reduce spam
This breaks azure bulk aggregation, but now the code structure is in place to support both
Tests: ran unit tests
Bugfix: AnalyticsV2 - don't miss client-only initialization during bootstrap flow
Tests: monitored client bootstrap analytics
Bugfix: AnalyticsV2 - skip client analytics if steam hasn't been started
Tests: none, trivial change
Bugfix: AnalyticsV2 - Log and DryRun no longer skip accumulators and uploaders once changed
As an extra, it got rid of a bunch of inheritance code
Tests: ran unit tests and logged analytics in editor
Clean: leaving a TODO note for one I return to this
Update: if either logging or dry run is enabled, log uploader activity
Previously it was editor only, and you had to enable dry-run specifically to get those logs. This makes it easier to work with.
Tests: none, trivial change
Bugfix: in editor with SERVER+CLIENT send client records to client gameplay AnalyticsTable
Not sure if it's technicall bug, but it did differ from original logic
Tests: logged activity in the editor
Optim: wind down AzureAnalyticsUploader when UseV2 is active
Gets rid of async tasks that are generating allocs while waiting for work
Tests: in editor activated v2, waited till upload happens, then disabled and waited for normal upload
Update: name the analytics thread
Tests: dry run in the editor
Optim: reduce allocations by relying less on async/await
- Use a Thread instead of a Task for internal AnalyticsManager work (this also fixed editor 10s shutdown delay)
- Allow internal upload tasks to run in parallel instead of waiting on sequential completion
Previously it would frequently GC, now it's back to normal
Tests: ran in editor in dry run mode. ran unit tests
Update: add profiling markers to AnalyticsManager and it's uploaders
Tests: ran in editor
Update: FPUploaderImpl - add a warning in case someone attempts to feed it non-json AnalyticsTable
Tests: none, trivial change
Update: append AnalyticsTable name when logging accumulator activity
Tests: none, trivial change
Update: accumulator log now also outputs byte size (pre compression if there is one)
Tests: none, trivial change
Bugfix: fix false "EventRecord pooling error" when disabling analytics.usev2
Hoping to rip it out once we fully convert to v2
Tests: unit tests + enabled-disabled in editor
Update: move analytics-for-server shutdown to ServerMgr.CloseConnection
This shuts down server before steamwroks in SERVER+CLIENT editor - makes editor's lifecycle management less confusing
Tests: confirmed client session end event still fires in editor
Bugfix: ensure session end event gets enqueued for V2 analytics
Tests: monitored in editor, saw it when stopping player
Clean: move new implementation into it's own file
Might split it up further, but not needed for now
Tests: editor compiles
Bugfix: respect rate limiters of AnalyticsTables and avoid mixing data when uploading
- Fixed by creating an accumulator per AnalyticsTable that's managed by uploaders
I need to profile and work on allocs, but it seems to be working
Tests: unit tests and monitored client telemetry
Update: Integrate the new AnalyticsManager into existing logic
- Guarded by `analytics.usev2` (default off for now)
- Bugfix for rate-limiting being disabled due to mixing Now and UtcNow (whoops)
- Added global `analytics.dryrun` and `analytics.log`
Did a cursory test in editor using dry running and it seems to be working, but it leaks performance data into player_tick blobs. Need to reorganize accumulators again - always-aggregate isn't always valid.
Tests: used above switches and monitored data flow.
Update: Add FallbackUploader - uses primary uploader if enabled, otherwise uses fallback
- Added a test that checks it
- DryRun is now a per-accumulator settings instead of a global overwrite - lets each uplaoder decide how to act
Allows us to setup flexible chains(extra loggers, backups, etc), and more importantly we can emulate current analytics flow
Tests: ran unit tests
Update: Move accumulators into uploaders
- AnalyticsTests.TestUploade has logging enabled permanently to help validate things
It was a blocker to implement uploader chains (which I discovered we need for server, but should be quick)
Tests: ran unit tests
Clean: rename AnalyticsUploader -> AnalyticsManager
Disambiguates from the internal IUploader
Tests: compiles in editor
Bugfix: apply separator in json-lists aggregation correctly
Tests: used accumulator logging to see what is being fed
Update: Accumulators can now log what they're being fed
This revealed a bug with json concatenation, should be a quick fix
Tests: ran unit tests
Update: AnalyticsUploader - combine AzureBulkJsonUploader and AzureBulkCSVUploader into BulkAccumulator
- this also allows to combine the 2 uploaders into 1
This enabled aligning EventRecord serialization with current system. I think this is the last change to the code, next up will updating user code and testing in "real world".
Tests: ran unit tests
Update: AnalyticsTable - bring back useJsonDataObject support
Tests: ran unit tests
Tests: add all Analytics Uploader tests
- Covers ClientFPUploader, ServerFPUploader, AzureBulkJsonUploader, AzureBulkCSVUploader with multithread data generation
- Specialized editor logging for each uploader, as it doesn't make sense to emit compressed goup
ClientFPUploader will need internet connection as it relies on steam
Tests: ran unit tests
Update: AnalyticsUploader - refactor how we accumulate data
- AnalyticsTable no longer serializes EventRecords, that's moved to IAccumulators
- Implement serializers on top of IAccumulators (Csv, Json, CompressedCsv, CompressedJson)
- Uploaders automatically create IAccumulators per AnalyticsTable
Still missing an accumulator that would replicate how client analytics are serialized, but now it should be easy to do. Unit tests got simplified, need to expand coverage
Tests: ran unit tests
Update: implement AzureBulkUploaderImpl
- uses UploadBlobAsync instead of OpenWriteAsync
Not tested. Realized that current abstractions will not play well together, so gotta move a bit of stuff around.
Tests: none, will do later once I start converting user-code
Update: AnalyticsUploader - implement FPUploaderImpl
- add editor-only DryRun switch to log everything instead of actually sending it
- IUploader.Upload is now async
Tests: none, not hooked up to anything yet, that'll be next experiment
Update: AnalyticsTable - return EventRecord back to pool
- amended test to check for pool usage
Tests: ran unit tests
Update: AnalyticsUploader now flushes it's queue on shutdown.
- Updated tests to validate everything got flushed
- Fixed missed gzipstream end block serialization tracking
- Fixed ever-growing memory stream
Next up need to add EventRecord pooling, then gotta implement remaining uploaders, and in theory it's done
Tests: ran unit tests
Update: add per-AnalyticsTable telemetry
- using this telemetry to improve unit tests
Next up need to add support for flush-on-shutdown
Tests: ran unit tests
Update: Initial rewrite of the analytics core
- Only implements NullUploader, I'll implement actual uploaders after tests are good
- Added basic tests for it, but they don't validate anything - that'll be next.
- Renamed existing AnalyticsTests.cs to Test.EventRecord.cs (+ internal rename) - those tests validate EventRecord serialization
Goal is to use one implementation to cover needs of both Client and Server. Got the basics done, need to add telemetry to be able to test better, support EventRecord pooling and implement the existing uploaders.
Tests: ran new unit tests
