Consider changing to use memory pools

[svermeulen]

There are a lot of small memory allocs happening in a lot of areas. We avoid a lot of these allocs by using a static memory pool