Don't use MarkDirty in collider, it is causing a double collider rebuild, in turn causing rigidbody start asleep to not work