Since BufferedHashSet is more expensive now, lets try to avoid it where we don't need the concurrency