61 Commits over 335 Days - 0.01cph!
▅▅▍▉█ ▌▌▆█▉█▊▌▉▌▉▆▅█▅▇▋▉▊▉▌▊▇▊█▍▇ ▉▅▍█▊▌█▄▆▄ ▌█▄ ▉▌▊ ▌▆
▊▆▍▍▄▆█▋▄.█▅▆▋▍▋ -> ▊▍▇▇.▄▍▄▄▉▆
▍▄▇ ▍█▋▍ ▊▊ ▅▉███ ▅▅▆▍▉▅.▆▅▆▌█▌▄▅▄█▉ ▆▅▇ ▅▅▍ ▋▅███ ▄▍▋▆ ▆▅▌▅▇▉▉▉▆▌▋▉ █▉▋▌▊ ▅▉▇▆▅.▆▄▆▉▌▄▋▋▄
Bugfix: reset stream position after copying data
- Backups was using a stream with a cursor at the end, so it was sending 0-length content
Tests: ran in test env, was able to download non-empty files
Merge pull request #8 from Facepunch/FixEmptyBackups
Bugfix: Send full backup files
Bugfix: reset stream position after copying data
- Backups was using a stream with a cursor at the end, so it was sending 0-length content
Tests: ran in test env, was able to download non-empty files
New: support xml-style errors from API
Allows to see more details about what's going wrong with the request(Azure SAS errors are xml)
Tests: got the SAS error locally, saw the logs in ServerManager
Bugfix: don't use a closed stream when retrying backup uploading
If we fail the upload at the point of consuming the stream (so passed backupBegin), it will close the stream and fall over on Position call, making retries useless.
Tests: when running own server it was erroring out on retries with stream closed exception - not anymore.
Bugfix: Don't wait after last upload retry
Tests: Saw the uploads fail, didn't see the log after last attempt.
Update: renamed file back to BackupData
I shouldn't have renamed it in the first place.
Clean: removing obsolete property
Tests: local run was able to upload files
Bugfix: respect the cancellation token when fetching a response of request
Update: limit how much time upload routine can take
- also add extra logging around processGuard semaphore
Tests: tested the entire flow by triggering 5 snapshots
Bugfix: use the right token for some async ops
Bugfix: avoid unlockign semaphore from the wrong thread
If it would time out during the wait period, it would release the semaphore in finally, leading to break of unique ownership promise
Clean: remove ITask inheritance from UploadPerfSnapshot
It used to be a task at the start, but not anymore
Update: review feedback
- dispose of the nested CTS for perf snapshot uploads
- Use async copy when creating an extra stream to send backup files
Tests: full local server flow + 2 snapshot uploads
Merge pull request #7 from Facepunch/DebugImprovements
Minor fixes and more verbose HTTP errors
WIP restart server at 4am local machine time, check for >=22 hour server uptime and do not restart Staging servers (env || hostname)
start the timer 10 minutes after the server is up and repeat every 5 minutes
Separate rcon logic and serverinfo loop out of ConsoleInputBar and into separate class CommandManager. Swapped out usages in RunServer.cs. Finished up the CheckForServerRestart functionality. Confirmed the project still builds
Was accidentally using DateTime.UtcNow instead of DateTime.Now
Change the time check to prevent the extremely rare edgecase of the server attempting to restart twice
Use the global cancellation token in the CommandManager, don't make a new one
CommandManager takes CancellationToken, not source
Rename CommandManager to more suitable ServerRcon
rename vars, too
Double time window to ensure we restart and store DateTimeOffset of when restart was last requested
Attempt to restart multiple times if we failed due to an exception
Log any errors
Minor changes
Merge pull request #6 from Facepunch/server_restarts
Server restarts
Attempt to restart multiple times if we failed due to an exception
Double time window to ensure we restart and store DateTimeOffset of when restart was last requested
Rename CommandManager to more suitable ServerRcon
Update: review feedback
- dispose of the nested CTS for perf snapshot uploads
- Use async copy when creating an extra stream to send backup files
Tests: full local server flow + 2 snapshot uploads
Bugfix: avoid unlockign semaphore from the wrong thread
If it would time out during the wait period, it would release the semaphore in finally, leading to break of unique ownership promise
Clean: remove ITask inheritance from UploadPerfSnapshot
It used to be a task at the start, but not anymore
CommandManager takes CancellationToken, not source
Was accidentally using DateTime.UtcNow instead of DateTime.Now
Change the time check to prevent the extremely rare edgecase of the server attempting to restart twice
Use the global cancellation token in the CommandManager, don't make a new one
Bugfix: use the right token for some async ops
Update: limit how much time upload routine can take
- also add extra logging around processGuard semaphore
Tests: tested the entire flow by triggering 5 snapshots
Bugfix: respect the cancellation token when fetching a response of request
New: support xml-style errors from API
Allows to see more details about what's going wrong with the request(Azure SAS errors are xml)
Tests: got the SAS error locally, saw the logs in ServerManager
Bugfix: don't use a closed stream when retrying backup uploading
If we fail the upload at the point of consuming the stream (so passed backupBegin), it will close the stream and fall over on Position call, making retries useless.
Tests: when running own server it was erroring out on retries with stream closed exception - not anymore.
Bugfix: Don't wait after last upload retry
Tests: Saw the uploads fail, didn't see the log after last attempt.
Update: renamed file back to BackupData
I shouldn't have renamed it in the first place.
Clean: removing obsolete property
Tests: local run was able to upload files
Separate rcon logic and serverinfo loop out of ConsoleInputBar and into separate class CommandManager. Swapped out usages in RunServer.cs. Finished up the CheckForServerRestart functionality. Confirmed the project still builds
start the timer 10 minutes after the server is up and repeat every 5 minutes
WIP restart server at 4am local machine time, check for >=22 hour server uptime and do not restart Staging servers (env || hostname)
New: Find and upload perf snapshots
- Uploads as one request, rather than a ping-pong chain
- Once attempted the upload, deletes from the drive
- Reacts to log messages
I've yet to test it all, will do after done with Dashboard
Update: Renaming BackupBeginResult -> DelegatedUploadBeginResult
It's renamed on the dashboard, and will be used for 2 flows, so more fitting
Update: reimplementing perf snapshot uploading to use delegation
Merge: from main
Update: review comments
- Removed in-memory file caching and simplified code
- Added a semaphore to prevent multiple concurrent uploads
- Removed a flat delay when detected trigger phrase to upload a snapshot
- Removed post-heartbeat attempt to find-and-upload snapshots
Update: braking change mitigation
- Fallback to old API json format if failing to process with new one.
Update: further review feedback
- Wrapped body of FindAndUpload in a try-finaly to ensure we always release a semaphore
- Merged backwards compatibility support into existing DelegatedUploadBeginResult class
Clean: file rename (since it no longer contains old class)
Merge pull request #5 from Facepunch/PerfSnapshot
Adding support for PerfSnapshot uploads to Dashboard
Update: further review feedback
- Wrapped body of FindAndUpload in a try-finaly to ensure we always release a semaphore
- Merged backwards compatibility support into existing DelegatedUploadBeginResult class
Clean: file rename (since it no longer contains old class)
Update: review comments
- Removed in-memory file caching and simplified code
- Added a semaphore to prevent multiple concurrent uploads
- Removed a flat delay when detected trigger phrase to upload a snapshot
- Removed post-heartbeat attempt to find-and-upload snapshots
Update: braking change mitigation
- Fallback to old API json format if failing to process with new one.
Add a fork of CliWrap here that adds a way to access the Process instance to set affinity
Automatically set server process affinity when running on a CPU with split 3D v-cache
Increase timeout for running rcon commands from 10s to 30s
Merge: from main
Update: Renaming BackupBeginResult -> DelegatedUploadBeginResult
It's renamed on the dashboard, and will be used for 2 flows, so more fitting
Update: reimplementing perf snapshot uploading to use delegation
Add a fork of CliWrap here that adds a way to access the Process instance to set affinity
Automatically set server process affinity when running on a CPU with split 3D v-cache
Increase timeout for running rcon commands from 10s to 30s
New: Find and upload perf snapshots
- Uploads as one request, rather than a ping-pong chain
- Once attempted the upload, deletes from the drive
- Reacts to log messages
I've yet to test it all, will do after done with Dashboard
Check for and apply Harmony mod updates automatically while the server is running
Backup save file right after the server saves
Remove redundant cancellation token source in server manager
Add some more guards to try and prevent exiting out when not actually cancelled
Gracefully stop the Rust server when closing server manager with Ctrl+C
More logging to troubleshoot missing heartbeat issues
Workaround for issues on even older windows console hosts?
Fix backspace with empty buffer exception
Fix unknown commands not clearing the command input
Workaround status bar issues when running under the old windows console host
Refactor logging so it doesn't go through Console
Add support for status bars at the bottom of the terminal
Implement status bar to connect to the server's rcon to show performance info and allow running commands
Set the -parent-pid command line argument when running the Rust server
Update files before backing up server data so we don't double backup when there is a server manager update
Switch to using new backup endpoint
Fix exception in heartbeat worker when exiting with Ctrl+C
Detect server crashes and make sure it restarts
Proper timer to make sure logs are submitted after some time instead of waiting for a full batch
Make the heartbeat loop more reliable and auto-restart if it fails
▄▋▊▄▌▌ ▊▆█ ▆▌▋ ██▊▅▅▉▇▊ ▆▍▄▄▅
