userDaniel Pcancel
branchrust_reboot/main/analytics_queue_rewritecancel

44 Commits over 30 Days - 0.06cph!

Yesterday
Merge: from main
Yesterday
▊▊▉▇▋▇: ▅▅▅▆▅█▊▌▉▊▉ - ▆▉▍▅▄▋ ▍█▆▇ ▆▉▉▋▄▊▇ ▆▊▉█ ▍▌▇▍▄▊▅▍▊▆▄▆▌▉▉▆▄ ▉▍▊▍ ▉▄▌▊ ▊▉▅▌▇▊ ▌▄▅ ▇▋▉▆▄▅-▋▉▊▅▆▆▊▅▉ █▆▊▋▍▉▇▋▉ ▆▇ ▄▍-▅▇▊ ▅▅▋ ▆▋▊▌█▌ ▌▇▌▍▉▇▍ ▇▊▇ ▄▍▄▅ ▊▍▍▄▍▊▄▋ ▇▇ █▄▅▌▍▋▌ █▆▌▄▅: ▉▌▍▍▇▋▇ ▋▆▋▄ ▄▅▇▌▉▌, ▋▌▄▄▉██▆▄ █▍▉▊ ▊▆▄▉█▄▄ ▄▇▆ █▉▄▋▄▄▋▊/▍▆▉█▄▍▄▍██▋
Yesterday
Update(editor): AnalyticsV2 - emit session start event This is to make validating analytics on the backend easier (as this helps to form a full session) Tests: with logging enabled observed session-start being fired before player_connect for server is sent
Yesterday
Update: AnalyticsV2 - add support for changing upload intervals - made RuntimeProfiler and TickLogger adjust their intervals Tests: enabled runtime profiler and changed interval - confirmed sent when expected
Yesterday
Update: AnalyticsV2 - restore gameplay_events upload interval Accidentally submitted while testing it yesterday Tests: none, trivial change
Yesterday
Bugfix: AnalyticsV2 - don't spam "appended 0 bytes" when logging is enabled Tests: none, trivial change
Yesterday
Bugfix: AnalyticsV2 - ensure events sent at shutdown have valid steam tickets Done by pumping steam event queue while analytics thread is shutting down Tests: unit tests and ran Client mode in editor
Yesterday
Bugfix: AnalyticsV2 - only recycle auth tickets if not consumed them - Added extra checks for early steam shutdowns Previously we would eagerly cancel tickets, which is not api-appropriate. Now only doing so if either an exception occured or we're in dry-run mode Tests: ran unit tests. Still can't catch why auth ticket is sometimes null on shutdown
5 Days Ago
Bugfix: AnalyticsV2 - fix invalid Json aggregation - logs are aggregated again to reduce spam This breaks azure bulk aggregation, but now the code structure is in place to support both Tests: ran unit tests
5 Days Ago
Bugfix: AnalyticsV2 - don't miss client-only initialization during bootstrap flow Tests: monitored client bootstrap analytics
5 Days Ago
Bugfix: AnalyticsV2 - skip client analytics if steam hasn't been started Tests: none, trivial change
5 Days Ago
Bugfix: AnalyticsV2 - Log and DryRun no longer skip accumulators and uploaders once changed As an extra, it got rid of a bunch of inheritance code Tests: ran unit tests and logged analytics in editor
5 Days Ago
Merge: from main
16 Days Ago
Clean: leaving a TODO note for one I return to this
19 Days Ago
Update: if either logging or dry run is enabled, log uploader activity Previously it was editor only, and you had to enable dry-run specifically to get those logs. This makes it easier to work with. Tests: none, trivial change
19 Days Ago
Bugfix: in editor with SERVER+CLIENT send client records to client gameplay AnalyticsTable Not sure if it's technicall bug, but it did differ from original logic Tests: logged activity in the editor
19 Days Ago
Optim: wind down AzureAnalyticsUploader when UseV2 is active Gets rid of async tasks that are generating allocs while waiting for work Tests: in editor activated v2, waited till upload happens, then disabled and waited for normal upload
19 Days Ago
Update: name the analytics thread Tests: dry run in the editor
19 Days Ago
Optim: reduce allocations by relying less on async/await - Use a Thread instead of a Task for internal AnalyticsManager work (this also fixed editor 10s shutdown delay) - Allow internal upload tasks to run in parallel instead of waiting on sequential completion Previously it would frequently GC, now it's back to normal Tests: ran in editor in dry run mode. ran unit tests
19 Days Ago
Update: add profiling markers to AnalyticsManager and it's uploaders Tests: ran in editor
19 Days Ago
Update: FPUploaderImpl - add a warning in case someone attempts to feed it non-json AnalyticsTable Tests: none, trivial change
19 Days Ago
Update: append AnalyticsTable name when logging accumulator activity Tests: none, trivial change
19 Days Ago
Update: accumulator log now also outputs byte size (pre compression if there is one) Tests: none, trivial change
19 Days Ago
Bugfix: fix false "EventRecord pooling error" when disabling analytics.usev2 Hoping to rip it out once we fully convert to v2 Tests: unit tests + enabled-disabled in editor
19 Days Ago
Update: move analytics-for-server shutdown to ServerMgr.CloseConnection This shuts down server before steamwroks in SERVER+CLIENT editor - makes editor's lifecycle management less confusing Tests: confirmed client session end event still fires in editor
19 Days Ago
Bugfix: ensure session end event gets enqueued for V2 analytics Tests: monitored in editor, saw it when stopping player
19 Days Ago
Clean: move new implementation into it's own file Might split it up further, but not needed for now Tests: editor compiles
19 Days Ago
Bugfix: respect rate limiters of AnalyticsTables and avoid mixing data when uploading - Fixed by creating an accumulator per AnalyticsTable that's managed by uploaders I need to profile and work on allocs, but it seems to be working Tests: unit tests and monitored client telemetry
20 Days Ago
Update: Integrate the new AnalyticsManager into existing logic - Guarded by `analytics.usev2` (default off for now) - Bugfix for rate-limiting being disabled due to mixing Now and UtcNow (whoops) - Added global `analytics.dryrun` and `analytics.log` Did a cursory test in editor using dry running and it seems to be working, but it leaks performance data into player_tick blobs. Need to reorganize accumulators again - always-aggregate isn't always valid. Tests: used above switches and monitored data flow.
20 Days Ago
Update: Add FallbackUploader - uses primary uploader if enabled, otherwise uses fallback - Added a test that checks it - DryRun is now a per-accumulator settings instead of a global overwrite - lets each uplaoder decide how to act Allows us to setup flexible chains(extra loggers, backups, etc), and more importantly we can emulate current analytics flow Tests: ran unit tests
20 Days Ago
Update: Move accumulators into uploaders - AnalyticsTests.TestUploade has logging enabled permanently to help validate things It was a blocker to implement uploader chains (which I discovered we need for server, but should be quick) Tests: ran unit tests
20 Days Ago
Clean: rename AnalyticsUploader -> AnalyticsManager Disambiguates from the internal IUploader Tests: compiles in editor
20 Days Ago
Bugfix: apply separator in json-lists aggregation correctly Tests: used accumulator logging to see what is being fed
20 Days Ago
Update: Accumulators can now log what they're being fed This revealed a bug with json concatenation, should be a quick fix Tests: ran unit tests
21 Days Ago
Update: AnalyticsUploader - combine AzureBulkJsonUploader and AzureBulkCSVUploader into BulkAccumulator - this also allows to combine the 2 uploaders into 1 This enabled aligning EventRecord serialization with current system. I think this is the last change to the code, next up will updating user code and testing in "real world". Tests: ran unit tests
21 Days Ago
Update: AnalyticsTable - bring back useJsonDataObject support Tests: ran unit tests
21 Days Ago
Tests: add all Analytics Uploader tests - Covers ClientFPUploader, ServerFPUploader, AzureBulkJsonUploader, AzureBulkCSVUploader with multithread data generation - Specialized editor logging for each uploader, as it doesn't make sense to emit compressed goup ClientFPUploader will need internet connection as it relies on steam Tests: ran unit tests
21 Days Ago
Update: AnalyticsUploader - refactor how we accumulate data - AnalyticsTable no longer serializes EventRecords, that's moved to IAccumulators - Implement serializers on top of IAccumulators (Csv, Json, CompressedCsv, CompressedJson) - Uploaders automatically create IAccumulators per AnalyticsTable Still missing an accumulator that would replicate how client analytics are serialized, but now it should be easy to do. Unit tests got simplified, need to expand coverage Tests: ran unit tests
21 Days Ago
Update: implement AzureBulkUploaderImpl - uses UploadBlobAsync instead of OpenWriteAsync Not tested. Realized that current abstractions will not play well together, so gotta move a bit of stuff around. Tests: none, will do later once I start converting user-code
22 Days Ago
Update: AnalyticsUploader - implement FPUploaderImpl - add editor-only DryRun switch to log everything instead of actually sending it - IUploader.Upload is now async Tests: none, not hooked up to anything yet, that'll be next experiment
22 Days Ago
Update: AnalyticsTable - return EventRecord back to pool - amended test to check for pool usage Tests: ran unit tests
22 Days Ago
Update: AnalyticsUploader now flushes it's queue on shutdown. - Updated tests to validate everything got flushed - Fixed missed gzipstream end block serialization tracking - Fixed ever-growing memory stream Next up need to add EventRecord pooling, then gotta implement remaining uploaders, and in theory it's done Tests: ran unit tests
22 Days Ago
Update: add per-AnalyticsTable telemetry - using this telemetry to improve unit tests Next up need to add support for flush-on-shutdown Tests: ran unit tests
23 Days Ago
Update: Initial rewrite of the analytics core - Only implements NullUploader, I'll implement actual uploaders after tests are good - Added basic tests for it, but they don't validate anything - that'll be next. - Renamed existing AnalyticsTests.cs to Test.EventRecord.cs (+ internal rename) - those tests validate EventRecord serialization Goal is to use one implementation to cover needs of both Client and Server. Got the basics done, need to add telemetry to be able to test better, support EventRecord pooling and implement the existing uploaders. Tests: ran new unit tests