branchrust_reboot/main/analytics_queue_rewritecancel
10 Commits over 30 Days - 0.01cph!
Update: AnalyticsUploader - combine AzureBulkJsonUploader and AzureBulkCSVUploader into BulkAccumulator
- this also allows to combine the 2 uploaders into 1
This enabled aligning EventRecord serialization with current system. I think this is the last change to the code, next up will updating user code and testing in "real world".
Tests: ran unit tests
Update: AnalyticsTable - bring back useJsonDataObject support
Tests: ran unit tests
Tests: add all Analytics Uploader tests
- Covers ClientFPUploader, ServerFPUploader, AzureBulkJsonUploader, AzureBulkCSVUploader with multithread data generation
- Specialized editor logging for each uploader, as it doesn't make sense to emit compressed goup
ClientFPUploader will need internet connection as it relies on steam
Tests: ran unit tests
Update: AnalyticsUploader - refactor how we accumulate data
- AnalyticsTable no longer serializes EventRecords, that's moved to IAccumulators
- Implement serializers on top of IAccumulators (Csv, Json, CompressedCsv, CompressedJson)
- Uploaders automatically create IAccumulators per AnalyticsTable
Still missing an accumulator that would replicate how client analytics are serialized, but now it should be easy to do. Unit tests got simplified, need to expand coverage
Tests: ran unit tests
Update: implement AzureBulkUploaderImpl
- uses UploadBlobAsync instead of OpenWriteAsync
Not tested. Realized that current abstractions will not play well together, so gotta move a bit of stuff around.
Tests: none, will do later once I start converting user-code
Update: AnalyticsUploader - implement FPUploaderImpl
- add editor-only DryRun switch to log everything instead of actually sending it
- IUploader.Upload is now async
Tests: none, not hooked up to anything yet, that'll be next experiment
Update: AnalyticsTable - return EventRecord back to pool
- amended test to check for pool usage
Tests: ran unit tests
Update: AnalyticsUploader now flushes it's queue on shutdown.
- Updated tests to validate everything got flushed
- Fixed missed gzipstream end block serialization tracking
- Fixed ever-growing memory stream
Next up need to add EventRecord pooling, then gotta implement remaining uploaders, and in theory it's done
Tests: ran unit tests
Update: add per-AnalyticsTable telemetry
- using this telemetry to improve unit tests
Next up need to add support for flush-on-shutdown
Tests: ran unit tests
Update: Initial rewrite of the analytics core
- Only implements NullUploader, I'll implement actual uploaders after tests are good
- Added basic tests for it, but they don't validate anything - that'll be next.
- Renamed existing AnalyticsTests.cs to Test.EventRecord.cs (+ internal rename) - those tests validate EventRecord serialization
Goal is to use one implementation to cover needs of both Client and Server. Got the basics done, need to add telemetry to be able to test better, support EventRecord pooling and implement the existing uploaders.
Tests: ran new unit tests
