This would cost some memory, but on the other hand it would reduce a lot of load on the GC. We could also cache, e.g., to_bytes on GriddedPerms and maybe some symmetries.