What combinations of architectural optimization techniques have been employed in current state-of-the-art large language models (LLMs)? Where can I find detailed information on this topic?

FlashAttention RetNet RWKV Mamba/Mamba2 Hyena LFM ...