Don't transform rigidbody physics on interp transform change (why are we calling this?) we only want to directly set physics body transform if someone directly changed the game object transform !! Move collider keyframe body move to fixed update