964 Commits over 243 Days - 0.17cph!
Update: ServerProfiler - emit working set as KB instead of Bytes
- Also pre-allocate extra size in string builder to account for the memory counters.
KB are easier to spot on the graph changing(not really on perfetto's Values view, better on delta view)
Tests: exported in editor on Craggy
Update: ServerProfiler - vizualize process working set memory
- using debug binaries for now(bc3e74cd)
Need to add virtual set as well and test standalone servers(Win and Linux)
Tests: in editor on craggy.
Bugfix: ProfilerExporter - filter out worker thread marks that are before the frame start
It was originally correct, but in previous commit I simplified the code, thus breaking it. This restores it.
Tests: none, as currently too many changes present in workspace - will test later
Update: ServerProfiler - emit fake "Allocation" threads and duplicate allocation marks there
- Also updated the whole buffer estimation to take into account these metadata marks
- Added support for naming and sorting thread tracks
Well, they do appear, but the thread_sort_index is ignored by perfetto(see https://github.com/google/perfetto/issues/555). Might finally bite the bullet and write a protobuf exporter, but afraid that it'll be more expensive to run and won't compress as well.
Tests: exported a snapshot from editor
Update: ServerProfiler - enable safety checks by default
- Also prefixed error logs to make it easy to identify where it came from
Tests: did 6 exports in editor, no false-positives
Bugfix: ServerProfiler - don't emit thread tracks with only allocations in them
- Handle "legal" case where we don't have any method marks on worker threads after frame start timestamp
- Handle "legal" case where we get a thread profile for a thread that was stopped before the frame start
- Handle "legal" case where we get empty thread profiles due to method filtering
- Dead func removal
This can be a controversial choice, as allocations do happen there, but it's not something we can interact with because there's not enough helpful information about them(for example, what if we filter out entire thread methods?).
Tests: did 4 exports, wasn't able to find weird allocation records on different threads.
Update: ServerProfiler - Track allocations on all threads
The display of this information is still abysmal - need to figure out how to make it better.
Tests: took snapshots both in editor and in standalone server builds (win + linux). Hacked a version that used to crash, but current changes don't anymore.
Update: Impllement GamePhysics.HandleIgnoreCollisions as a batch
- extended TerrainCollision to support batching
- extended WaterColllision to support batching
There's still more improvements that can be done (translating into burst jobs and better job-graph building), but currently the goal is to translate more of code into batch form away from singular.
Tests: none, it's not hooked up yet - will explore writing unit tests next-ish
Bugfix: read back from the right array of heights after sampling ocean
- Reported by new unit tests
Tests: ran water-related unit tests - now they pass
Bugfix: Make sure WaterLevel tests run full simulation path during unit testing
Previously it was earlying out during GetWaterLevel
Tests: ran the new tests, they report an issue that I need to fix
Update: missing meta files from merge
Submitting for now, but should nuke once back on main
Tests: none, trivial change
Merge: from main
- Skipping a couple meta files for empty folders, will submit them separately
Tests: booted editor for import
Clean: removing unused WaterFactorForPlayer that didn't return WaterInfo
Tests: compiles in editor in SERVER+CLIENT mode
Bugfix: ensure we cache WaterInfo for players that use various vehicles
Some vehicles have custom logic to check how submerged the player is, and we would miss the water info taht was used to calculate it.
Tests: tested via DPV - submerged factor was 1 and waterinfo was valid (before it would report factor of 1, but invalid waterinfo).
Update: BaseMountable.WaterfactorForPlayer can now return WaterInfo
- allows further reusing of the same water info down the line
I actually need it to make sure all our player's cached values are correct - will fix in next submit.
Tests: none (as it's not in use in this CL)
Clean: fixing profiling scope name
Tests: none, trivial change
Update: FinalzieTickParallel now uses batched WaterFactors query
- had to add more fork-join-style code to deal with batching
This allows us to sample water information for all relevant players as a burst job (and there's further untapped potential). Next up is trying to batch and jobify physics checks.
Tests: played in a solo session with parallel processing enabled - running, driving, swimming worked normal. Played back all of my server demos with analysis enabled - no new violations detected.
Bugfix: use the right allocator for a waterlevels batch
- This revealed that we have an Application.isPlaying check that prevented this from being detected via unit tests
Tests: ran unit tests (but they're useless in this case), ran all demos that use the new code(yet to submit) - no more exceptions
Update: able to get WaterFactors for players as a batch operation
- internally splits players into sequential queries(vehicles, parenting) and a batch query
Another building block covered, I should be able to rewrite caching of water factors step of FinalizeMove(next submit).
Tests: just editor compilation, I need to adapt rest of code to work with it.
Update: BufferList - new utility methods
- Resize - to explicitly control internal capacity
- ContentSpan, ContentReadOnlySpan - shortcuts to create a ro-/span over the active part of the buffer
Need both as I'm using it further in batched player processing operations
Tests: none, trivial changes.
Bugfix: setup water query params fully
0s are valid, but it caused the sampling positions to lean towards origin
Tests: rerun the WaterLevel tests
Clean: removing unnecessary logging
Tests: none, trivial change
Update: Able to grab WaterInfos as a batch
- Added a test to validate it returns same results as non-batched
- Made sure WaterLevel tests work in an initialized scene
It's not fully converted to a batch - there are a couple calls internally that are yet to be converted (game physics and the like).
Tests: ran the WaterLevel tests
Update: WaterLevel supports getting water heights for a batch of positions
- added a test to make sure batched produces same results as the singular version
Tests: ran the new unit tests
Update: Putting in a skeleton of logic for Parallel update
- Duplicated BasePlayer.ServerUpdate and FinalizeMove - rewrote to work on a list of players
- Added `server.useplayerupdatejobs` to control whether to use parallel or serial processing of players, default to off
New code doesn't really do much, just organized in a way to start working on parallel bits. I feel like I'm about to depart on a year-long journey into the jungle.
Tests: compiled in editor in SERVER-CLIENT mode
Update: ServerDemoPlayer - use time as progress indicator, instead of bytes of stream consumed
- Also removed a couple TODOs that are either done or decided to drop
In the previous version there would be no progress report until the next timestamped messages was reached - this is confusing UX.
Tests: played back the demo where I connect 15 seconds in, causing a jump from 0% to 18% progress. Now it's gradual.
Update: DemoServer - don't try to launch it in SERVER+CLIENT mode
That's a can of worms I don't want to deal with right now
Tests: none, trivial change
Clean: DemoServer - allow configuring the server for demo playback
Tests: none, trivial change
Update: ServerDemoPlayer - added error logging on hierarchy desync
Since some messages are handled by hand, it's possible things can go wrong in the future - this should catch it.
Tests: played all demos I have, no errors reported
Clean: removing no longer needed method
- Was previously part of ClientDemoPlayer
Tests: builds in editor in SERVER mode
Bugfix: ServerDemoPlayer - handle player rotation properly
Tests: played back a new vehicle demo, and an old demo 2p where the player was shot in the head - no violations, no entity stutters
Bugfix: ServerDemoPlayer - replicated polayer now follows demo's trajectory
- Also added more detialed logging of position messages
Decided to stop avoiding syncing player's position to outgoing messages, as there's weird teleport in the recorded demo. But at least now it works as expected.
Tests: Played back the longer 2 player demo - all good. Playerd back the known short demo with violations - still present, so also good.
Bugfix: ServerDemoPlayer - don't duplicate connected players
- Also added expanded tick logging
Connected player doesn't move due to tick divergence - investigating.
Tests: played back a 2 player demo with 2nd player connecting half-way-through. Only one player gets created in expected position
▅▇█▋▉█: ▌▋▉▇▌▋█▋▍▋▅▆▌▇▌▇ - ▋█▋'▍ ▇▇▋▉▍▆▊ ▆▇█ ▆▉▋▉▉▉▌
- ▇▍▅▆▉ ▇▍▌▊▌ ▄▇▌▉▄▍ ▊▄█▍▍█▊▅▅▇ ▉█▊ ▉▇▋ ▇█ █▉▊▋▌▄▌▊ ▆█▊▍█▍▆
▉▅▉▅▉ ▍▌▆ ▄▄ ▆▌█ ▆▄▉▆▍█▄ █▇▌▅▇▄▆▅▆
█▆▌▉▅: ▄█▌▊▋█ ▉▆▊▊ ▊ ▆▇█▆ ▌█▍▄▌ ▋ ▆▆▇▋█▋ ▋▄▉▌▇▆ ▆▆▉▋▊▆▄█. ▆▊▆▉▄▋ ▋▌▇▄'▆ ▌▋█▌▆ █▌ ▆▊▌ ▅▄▌▇▄ ▄▆▊, ▅▌▋ ▍▇ ▉▉▍▋ ▋▋▍▌▊▉▉██▌
Update: ServerDemoPlayer - automatically validate the demo offsets if debug.json is available
Tests: Recorded a single player session on craggy - no reported errors.
Update: FullServerDemos - add basic debug json packet logging
I confirmed that my packet reconstruction goes wrong at some point, so now need this to help track down what logic is borked.
Tests: recorded a short demo on craggy - got a valid debug json
Bugfix: ServerDemoPlayer - quetly skip unrecognized entity kill messages
- Moved message logging to be before message handling, to be able to get more info before executing logic
Not clear why it's happening, but will leave that for later. Also investigating why getting malformed OnEntities messages once 2nd player connects.
Tests: tried to play back a 2 player demo on Craggy. No more NREs related to entity killing
Bugfix: DemoServer - properly cleanup outgoing NetWrites
Tests: played back a short server demo - no issues.
▅▆▋▆▊▆: █▍▄█▆▆█▄▄▊▅▍▉▉▄▋ - ▆▊▇▍█▅█▅ ▍▌ ▉▆▉▊▆▉ ▌▇▄▍▄▉ ▅ ▌▌▅▍▅▊ ▌▋▄▌▇ ▄▅▌▋▋▇ ▉▍ █▇▊▅/▉██▊▅▆ ▉▅ ▋▊█▉▅▆▅.
- ▆█▄▅ ▉█ ▅▊▅▊ ▅▉▊▇ ▅▄▌▅▊ ▊▉▋▄▋▉ ▉▋▊▇ ▊▇ ▉▉▆▇ ██ ▅▅▋ ▇▋▇▍ ▄▊▌▋ ▉▊ ▊ ▇▉▊/▉▉█▇▊ ▍▊█ ▇█▇▉ ▄█▄▊▊▊ ▄▌▆▋▇▄▋.
▌▊▄▄▊: ▄▍▄▌▊▌ ▍▄▉▅ ▄ ▌ █▌▉█▉▇ ▊▍▍▍ ▆▊█▋▅ ▇ ▊▋▆▊█▊█▇▉ ▆▌▆▋▉▄▌▋▇▇ █▉ ▆▇▇ ▇▅▊▄▋ █ ▇▌▊▊█▌▌. ██▋█▆▍▆▉█ ▆▇▍█ ▌▄█ ▊ ▌▇▇▍ ▆▆▍█▊ ▄▍ ▍▉▄▅▆▍ ▇▍▉ ▄▆▇▄▋▍▍▆▄ ▆▊▇█▄▋▅.
Update: ServerDemoPlayer - Only maintain entities from demo stream
- Destroys any entities that server decides to create, if they were not part of demo stream
- Reserve a 100k network IDs when playing a demo to avoid ID clashes
- Refactored message logging when playing back demos
Although not robust, this allows us to simulate a core set of functionality. Will need massaging to expand further.
Tests: played back 4 separate Craggy demos - all played back properly.
Update: ServerDemoPlayer - add debug option to log packets
Trying to figure out why some construction events are not being palyed back properly
Tests: played back a demo where I'm building multiple chained staircases
Bugfix: ServerDemoPlayer - synthesize initial tick after loadign a player
Avoids server's perception of teleport, as ticks are delta serialized and after load of save have default values.
Tests: played back a demo on craggy where I stayed stationary the first 10 seconds. No longer generating errors.
Bugfix: FullServerDemos - ensure initial save always happens if demo thread was toggled
Tests: recorded 2 demos in 2 recording sessions on one server session - both had saves created
Bugfix: ServerDemoPlayer - wake up all recognized players
Players that are loaded from a save arrive sleeping, which prevents tick processing.
Tests: Played back a couple short demos where I didn't jump at the start - saw player move.
Update: DemoServer - properly dispose resources on playback end
Tests: played back a server demo - no new errors
Merge: from main
Tests: none
Update: new ProtobufCodegen bins + ran generation
Moved my utility funcs for reading u64 from buffers from Rust project to the library itself
Tests: played back craggy demo
Buildfix: remove no longer existing method
Tests: editor compiles in Client+Server
Clean: DemoServer - removing custom tick vizualization logic
It should be via IDemoAnalyzers, rather than some randomly sprinkled seasoning
Tests: editor compiled in Server mode
Update: ServerDemoPlayer - players now move based on tick history
- Added recording of userIds to full server demos (need to match against players in saves)
- Added a callback for when world loading is done to do any additional logic (in demo case, matching players)
Tests: played back a 30s demo from Craggy
