Make sure we don't instanciate PerceivedSmartobjectData or PerceivedUnitData more than once per index of the FixedArrays, and only on demand. After that, we just replace the internals.