Finally got around to reading Meta’s newly released LLaMA, a collection of “open” foundation models optimized for inference.

The key takeaway: if you build a moderately sized model and train it long enough on publicly available datasets, you can outperform models that are significantly larger and more expensive at inference. This challenges the assumption that bigger is always better when it comes to large language models.

While the “openness” is commendable, it’s not truly open if you restrict it to non-commercial use only and don’t release the model weights without an application process. That said, the architecture is still impressive and worth studying.

Check out the model card and repo on GitHub.