What happens between entering the prompt and seeing the first word appear
14 points
1/21/1970
2 days ago
by shbhmrzd
Comments
shbhmrzd
I have been writing a series on understanding LLMs from the ground up for software engineers.
This is the last part.
In this post I cover what happens when you type a prompt and hit enter. How the model generates one token at a time, why that is slow, what the KV cache does about it,and how decoding strategies like temperature, top-k, and top-p shape the response.
I have been writing a series on understanding LLMs from the ground up for software engineers. This is the last part.
In this post I cover what happens when you type a prompt and hit enter. How the model generates one token at a time, why that is slow, what the KV cache does about it,and how decoding strategies like temperature, top-k, and top-p shape the response.