branchrust_reboot/main/perf_test_fwkcancel
36 Commits over 61 Days - 0.02cph!
Update: Adding a wait statement
- Using it to help debug issues with procedural world not running world generation.
Tests: ran LoadProcTest - it waited
Update: Delete the symlink on end of test run
This was previously causing deletion of blobs on unity close/open
Tests: ran LoadProcTest couple times
Update: PerfFwkManager can switch scenes
- Done via a basic command interpreter (same as ConsoleSystem, just supporting coroutines)
- renamed PerfFwk.IsReady to IsDone
Tests: ran the LoadProcTest - asserts validated the switch is done, but no proc gen happening yet
Update: Executing deferred command and waiting for it's execution
- also fixed the deferred command not being executed because it's missing the prefix (this took weirdly long to spot)
Using it to pass control for the actual testing top the game assembly
Tests: Ran LoadProcTest, it passed
Hackfix: skip bootstrap smoke test at the start
- Key here is skipping error logs, since they trigger test failures.
Right now we build this weird mutant executable, and it tries to run server-side smoke test despite being on client. It emits an error, that we can't suppress that early, and which tips over the tests. I'll need to fix this properly once I got better build pipeline control.
Tests: ran LoadProcTest - it passed (though it's mostly a dummy test)
Buildfix: added missing namespace qualifier
Tests: none, trivial change
Bugfix: resolve location of asset bundles when running standalone perf test
- Also symlinking server bundles instead of client, for now
Tests: ran the LoadProcTest, no more logs of failing to open bundles
Merge: from main
Tests: none (big merge, going to fix any issues in follow up commits)
Update: changing starting scene to be Bootstrap when running a perf test
- Also removed direct world setup settings modification, instead pushing them to global World static params
We have to go via Bootstrap because that's what initializes global systems. Will need to massage platform-specific transition logic to account for perf testing workflow (need to support CLIENT+SERVER flow).
Tests: Test player fails because of CWD, but if I run TestPlayer manually it boots and loads to main menu, so I count it as pass-ish
Update: Symlink assetbundles on test player build
Tests: ran the test, validated that they got symlinked to expected path (but there are other issues, investigating)
Update: Sneak in console cmds via command line
Need a better way in the future, but for now this should allow to control test conditions.
Tests: none (need to fix assetbundle copying first), though validated it sohuld work by reading through code
Merge: from main
Tests: none
Add: Perf Test dud to boot ProcGen map
Builds for player and starts switching the world, but I try to set it up incorrectly and still need to build/copy asset bundles.
Tests: Confirmed that the player asserts when running the new test
Clean: removing unnecessary checks and files
Tests: none, trivial changes
Update: when enabling PerfFwk ensure we have 64bit arch selected
Somehow I had 32bit arch targetted locally, and it looks like this is a local-only setting, so it's possible others will also run into this - this should avoid issues (like previous problem with Rust.Harmony).
Tests: while on WIndows32 target enabled the framework - confirmed it switched to 64bit arch
Update: don't trample on existing defines when changing mode switches
Saves a bit of time when working with Performance Framework
Tests: confirmed RUST_PERF_FWK stays when switching to none, client, client+server
Buildfix: Harmony loader is conditionally built via asmdef settings
- Originally there was a mix of code macro checks and asmdef constraints - now it's just asmdef constraints and no code defines
- Enabled it for all platforms except editor, instead of just 3 explicit ones(Unity's TestFramework currently builds Win32 players instead of Win64).
Tests: with PerfFwk enabled and editor in different(Client, Server, both) modes, ran Pool perf tests in Player/standalone mode. Everything built and succeeded.
Update: Exclude unnecesary scenes when PerfFwk is enabled
- We have disabled scenes in the list that UnityTestFramework ignores and tries to build, leading to issues
- It also saves iteration time, since we only build scenes we'll use for perf testing
Tests: same as before
Update: Moving perf tests to PerfFwk plugin
- Added additional references to PerfFwk
Original idea of mixing perf test code into main assembly didn't work out, so for now going the path of contianing them in an isolated assembly
Tests: tested with other changes to run pooling tests in CLIENT+SERVER standalone mode
Undo: auto-reference of PerformanceTesting lib
Trying out a different approach
Tests: none
Merge: from main
Tests: none
Backout: bring back all the scenes
Without this, editor bootstrap workflow dies.
Tests: none, trivial change
Update: Reducing which scenes we have set for build in Build Settings
Our internal build macros only build 2 scenes, while Test Runner tries to build all (incl disabled) scenes set in the Build Settings. This allows us to run Player-mode tests without extra steps.
Tests: With PerfFwk enabled, ran pool perf tests in Player mode - it ran and gathered results.
Update: Reorganization of the perf test framework
- Unity.PerformanceTesting is marked as auto-referenced (this is another modification of original package)
- Added a "Rust Editor/Performance Framework/Active" menu toggle to enable/disable perf test scripts - controls RUST_PERF_FWK define
- Renamed and moved PerfFwk assembly to Plugins/ (as that's it's design)
- Moved existing perf tests outside to the root assembly (Scipts/PerfTests) so that we can access gameplay code
The goal is to both isolate the perf framework code from the codebase as much as possible(don't ship to players or unity devs to load stuff that's not sueful), while also being able to work with our main game scripts directly.
Tests:
- Switch the toggle on and off - no editor errors.
- Built client&server with framework being enabled - it passed.
-- Found no PerfFwk references in main game assembly for both client and server and no references to test classes
Buildfix: isolating perf test scripts into it's own assembly
- For the time being stores existing perf tests - will reorganize when the structure is more clear in the future.
Tests: Unity booted without errors, confirmed perf tests presence in Test Runner
Merge: from main
Tests: none
Update: output results in micros for Recursion test
Tests: ran the benchmark, confirmed output
Update: Adding a couple tail recursion perf tests
- also comes with a vailidity unit test
Going to be used to do a write up for the weekly perf blog post
Tests: Ran these perf tests in editor and player, got kind-of expected results.
Merge: from main
Tests: none
Update: Take marker telemetry every measurement
- Also avoid activating markers during Setup/Shutdown
- Also avoid trying to convert Undefined sampler type to "seconds"
Without these changes perf telemetry was too coarse(1 total value for all measurements * iters)
Tests: Ran it for existing bench tests
Bugfix: Don't record GC data from Warmup runs
Fallout from my first patch
Tests: ran new perf tests with warmup, observed no allocations being recorded(as expected)
Update: Switch to Dynamic measurements for Pool perf tests
- Also moved the tests into it's own nested class since there'll be more in the future.
Surprised this is not default behavior, but ah well. It does run less measurements, so it avoids random spikes.
Tests: ran the pool perf tests, results line up with previous
Bugfix: Avoid perf degradation with many small Perf Test iterations
Was caused by aggressive GC invocation, which I just ripped out - there's a better alternatives if it's truly needed.
Tests: Ran the Pool bench tests, execution time went down from 3m to 3-5s
Bugfix: Avoid tracking allocations from Setup/Cleanup
This fixes the bug that was invalidating measurement tracking. That said, this causes a perf regression because ExecuteSingleIteration is slow - will fix in next CL.
Tests: Observed a reduction in allocation calls for Pool perf test (but not to 0, as it's picking up allocations from GC.Collect)
New: Adding PoolTests allocation perf test
Estimates the allocation speed compared to basic heap alloc when we have full capacity available. This is just for perf testing package validation(revealed 2 bugs).
Tests: ran it and validated results in the Test Report window
Add: Adding Performance Test Framework(v3.0.3) as a source package
While evaluating I discovered a couple issues, so will be patching it over time
Tests: wrote a pool perf test, it ran
