Only deduct the population boost from destroyed buildings if they had finished construction to avoid overflow error