Only defer the destruction of building blocks, destroy all other entities immediately when leaving a networking group