branchrust_reboot/main/analytics_queue_rewritecancel

10 Commits over 30 Days - 0.01cph!

5 Hours Ago
Update: AnalyticsUploader - combine AzureBulkJsonUploader and AzureBulkCSVUploader into BulkAccumulator - this also allows to combine the 2 uploaders into 1 This enabled aligning EventRecord serialization with current system. I think this is the last change to the code, next up will updating user code and testing in "real world". Tests: ran unit tests
6 Hours Ago
Update: AnalyticsTable - bring back useJsonDataObject support Tests: ran unit tests
Today
Tests: add all Analytics Uploader tests - Covers ClientFPUploader, ServerFPUploader, AzureBulkJsonUploader, AzureBulkCSVUploader with multithread data generation - Specialized editor logging for each uploader, as it doesn't make sense to emit compressed goup ClientFPUploader will need internet connection as it relies on steam Tests: ran unit tests
Today
Update: AnalyticsUploader - refactor how we accumulate data - AnalyticsTable no longer serializes EventRecords, that's moved to IAccumulators - Implement serializers on top of IAccumulators (Csv, Json, CompressedCsv, CompressedJson) - Uploaders automatically create IAccumulators per AnalyticsTable Still missing an accumulator that would replicate how client analytics are serialized, but now it should be easy to do. Unit tests got simplified, need to expand coverage Tests: ran unit tests
Today
Update: implement AzureBulkUploaderImpl - uses UploadBlobAsync instead of OpenWriteAsync Not tested. Realized that current abstractions will not play well together, so gotta move a bit of stuff around. Tests: none, will do later once I start converting user-code
Yesterday
Update: AnalyticsUploader - implement FPUploaderImpl - add editor-only DryRun switch to log everything instead of actually sending it - IUploader.Upload is now async Tests: none, not hooked up to anything yet, that'll be next experiment
Yesterday
Update: AnalyticsTable - return EventRecord back to pool - amended test to check for pool usage Tests: ran unit tests
Yesterday
Update: AnalyticsUploader now flushes it's queue on shutdown. - Updated tests to validate everything got flushed - Fixed missed gzipstream end block serialization tracking - Fixed ever-growing memory stream Next up need to add EventRecord pooling, then gotta implement remaining uploaders, and in theory it's done Tests: ran unit tests
Yesterday
Update: add per-AnalyticsTable telemetry - using this telemetry to improve unit tests Next up need to add support for flush-on-shutdown Tests: ran unit tests
2 Days Ago
Update: Initial rewrite of the analytics core - Only implements NullUploader, I'll implement actual uploaders after tests are good - Added basic tests for it, but they don't validate anything - that'll be next. - Renamed existing AnalyticsTests.cs to Test.EventRecord.cs (+ internal rename) - those tests validate EventRecord serialization Goal is to use one implementation to cover needs of both Client and Server. Got the basics done, need to add telemetry to be able to test better, support EventRecord pooling and implement the existing uploaders. Tests: ran new unit tests