branchrust_reboot/main/perf_test_fwkcancel

28 Commits over 30 Days - 0.04cph!

9 Days Ago
Update: changing starting scene to be Bootstrap when running a perf test - Also removed direct world setup settings modification, instead pushing them to global World static params We have to go via Bootstrap because that's what initializes global systems. Will need to massage platform-specific transition logic to account for perf testing workflow (need to support CLIENT+SERVER flow). Tests: Test player fails because of CWD, but if I run TestPlayer manually it boots and loads to main menu, so I count it as pass-ish
9 Days Ago
Update: Symlink assetbundles on test player build Tests: ran the test, validated that they got symlinked to expected path (but there are other issues, investigating)
9 Days Ago
Update: Sneak in console cmds via command line Need a better way in the future, but for now this should allow to control test conditions. Tests: none (need to fix assetbundle copying first), though validated it sohuld work by reading through code
9 Days Ago
Merge: from main Tests: none
23 Days Ago
Add: Perf Test dud to boot ProcGen map Builds for player and starts switching the world, but I try to set it up incorrectly and still need to build/copy asset bundles. Tests: Confirmed that the player asserts when running the new test
23 Days Ago
Clean: removing unnecessary checks and files Tests: none, trivial changes
23 Days Ago
Update: when enabling PerfFwk ensure we have 64bit arch selected Somehow I had 32bit arch targetted locally, and it looks like this is a local-only setting, so it's possible others will also run into this - this should avoid issues (like previous problem with Rust.Harmony). Tests: while on WIndows32 target enabled the framework - confirmed it switched to 64bit arch
23 Days Ago
Update: don't trample on existing defines when changing mode switches Saves a bit of time when working with Performance Framework Tests: confirmed RUST_PERF_FWK stays when switching to none, client, client+server
23 Days Ago
Buildfix: Harmony loader is conditionally built via asmdef settings - Originally there was a mix of code macro checks and asmdef constraints - now it's just asmdef constraints and no code defines - Enabled it for all platforms except editor, instead of just 3 explicit ones(Unity's TestFramework currently builds Win32 players instead of Win64). Tests: with PerfFwk enabled and editor in different(Client, Server, both) modes, ran Pool perf tests in Player/standalone mode. Everything built and succeeded.
23 Days Ago
Update: Exclude unnecesary scenes when PerfFwk is enabled - We have disabled scenes in the list that UnityTestFramework ignores and tries to build, leading to issues - It also saves iteration time, since we only build scenes we'll use for perf testing Tests: same as before
23 Days Ago
Update: Moving perf tests to PerfFwk plugin - Added additional references to PerfFwk Original idea of mixing perf test code into main assembly didn't work out, so for now going the path of contianing them in an isolated assembly Tests: tested with other changes to run pooling tests in CLIENT+SERVER standalone mode
24 Days Ago
Undo: auto-reference of PerformanceTesting lib Trying out a different approach Tests: none
24 Days Ago
Merge: from main Tests: none
30 Days Ago
Backout: bring back all the scenes Without this, editor bootstrap workflow dies. Tests: none, trivial change
30 Days Ago
Update: Reducing which scenes we have set for build in Build Settings Our internal build macros only build 2 scenes, while Test Runner tries to build all (incl disabled) scenes set in the Build Settings. This allows us to run Player-mode tests without extra steps. Tests: With PerfFwk enabled, ran pool perf tests in Player mode - it ran and gathered results.
30 Days Ago
Update: Reorganization of the perf test framework - Unity.PerformanceTesting is marked as auto-referenced (this is another modification of original package) - Added a "Rust Editor/Performance Framework/Active" menu toggle to enable/disable perf test scripts - controls RUST_PERF_FWK define - Renamed and moved PerfFwk assembly to Plugins/ (as that's it's design) - Moved existing perf tests outside to the root assembly (Scipts/PerfTests) so that we can access gameplay code The goal is to both isolate the perf framework code from the codebase as much as possible(don't ship to players or unity devs to load stuff that's not sueful), while also being able to work with our main game scripts directly. Tests: - Switch the toggle on and off - no editor errors. - Built client&server with framework being enabled - it passed. -- Found no PerfFwk references in main game assembly for both client and server and no references to test classes
31 Days Ago
Buildfix: isolating perf test scripts into it's own assembly - For the time being stores existing perf tests - will reorganize when the structure is more clear in the future. Tests: Unity booted without errors, confirmed perf tests presence in Test Runner
31 Days Ago
Merge: from main Tests: none
51 Days Ago
Update: output results in micros for Recursion test Tests: ran the benchmark, confirmed output
53 Days Ago
Update: Adding a couple tail recursion perf tests - also comes with a vailidity unit test Going to be used to do a write up for the weekly perf blog post Tests: Ran these perf tests in editor and player, got kind-of expected results.
53 Days Ago
Merge: from main Tests: none
58 Days Ago
Update: Take marker telemetry every measurement - Also avoid activating markers during Setup/Shutdown - Also avoid trying to convert Undefined sampler type to "seconds" Without these changes perf telemetry was too coarse(1 total value for all measurements * iters) Tests: Ran it for existing bench tests
58 Days Ago
Bugfix: Don't record GC data from Warmup runs Fallout from my first patch Tests: ran new perf tests with warmup, observed no allocations being recorded(as expected)
59 Days Ago
Update: Switch to Dynamic measurements for Pool perf tests - Also moved the tests into it's own nested class since there'll be more in the future. Surprised this is not default behavior, but ah well. It does run less measurements, so it avoids random spikes. Tests: ran the pool perf tests, results line up with previous
59 Days Ago
Bugfix: Avoid perf degradation with many small Perf Test iterations Was caused by aggressive GC invocation, which I just ripped out - there's a better alternatives if it's truly needed. Tests: Ran the Pool bench tests, execution time went down from 3m to 3-5s
59 Days Ago
Bugfix: Avoid tracking allocations from Setup/Cleanup This fixes the bug that was invalidating measurement tracking. That said, this causes a perf regression because ExecuteSingleIteration is slow - will fix in next CL. Tests: Observed a reduction in allocation calls for Pool perf test (but not to 0, as it's picking up allocations from GC.Collect)
59 Days Ago
New: Adding PoolTests allocation perf test Estimates the allocation speed compared to basic heap alloc when we have full capacity available. This is just for perf testing package validation(revealed 2 bugs). Tests: ran it and validated results in the Test Report window
59 Days Ago
Add: Adding Performance Test Framework(v3.0.3) as a source package While evaluating I discovered a couple issues, so will be patching it over time Tests: wrote a pool perf test, it ran