What is SwiGLU?

from blog J. Carlos Roldán, | ↗ original
Most recent language models have switched from the typical ReLU and GELU activation functions to something called SwiGLU. In trying to understand it, I realized most materials assume a lot of prior knowledge and there isn't a simple explanation of why it is good. So let's go the bottom-up approach and try to explain it from scratch. Activation...