"Can We Understand How Large Language Models Reason?"
The Association of Computing Machinery (ACM) Posted a good article "Can We Understand How Large Language Models Reason?" about understanding what is known about LLM functionality and things we are learning. It's a short paper and the link is below. First some helpful background information.
How Large Language Models (LLMs) like Grok, ChatGPT, CoPilot, and Claude fit into the AI ecosystem:
AI → Machine Learning → Neural Networks → Deep Learning → Transformers → LLMs
Here's a quick parsing of the AI field to the subset of LLMs from Claude: "A quick way to keep it straight: not all AI is ML (a chess engine using brute-force search isn't "learning" anything). Not all ML is neural networks (a random forest predicting loan default risk isn't a neural network). Not all neural networks are "deep learning" in the modern sense (a small 2-layer network from the 1990s is technically a neural network but wouldn't be called deep learning today). And LLMs are one particular application of the transformer architecture, which itself is one particular way of building a deep neural network."
A couple of historical milestones for LLMs:
Eight Google scientists published the seminal paper on generative AI, transformers, and LLMs, "Attention Is All You Need", in June 12, 2017. The Wikipedia post is a short, good, and entertaining overview: https://en.wikipedia.org/wiki/Attention_Is_All_You_Need
OpenAI releases ChatGPT-3.5 on November 30, 2022
"Can We Understand How Large Language Models Reason?" https://cacm.acm.org/news/can-we-understand-how-large-language-models-reason/