Currently handles allocated as part of a HandleSet are stored in a linked list which is iterated every collection. We can do better.