LLM parameters, but what are they ?

No matter if you run local models or relying on frontier people are always talking about the number of parameters the model has. But what is this parameter and what does it hold ?

In this post I will try to share my learning experience to get a deeper view. If you recall from the previous post the input that’s passed towards the LLM is simply a matrix. This matrix consists of multiple vectors ( aka embeddings ), as each vector is a token.

There is one additional piece that we have to introduce before we focus on the parameters. The majority of LLMs have stacked up the number of transformers ( in Hugging face metadata the field is called ‘num_hidden_layers’ ). What happens to one of them is almost always the same to the others, so to simplify it we would have a look under just one. Also we imply that there is just a single head of attention to further simplify it. Basically a single stream inside which the user input passes.

Let’s say we have this simple interaction box with a user input and LLM land. Inside each transformer the tokens first must pass through the self attention layer. Here are the parameters that we are actually facing for the first time. Once our input matrix is inside the self attention block we are seeing those parameters for the first time.

Just to have a visual representation, this is the stage where we are. We are prompting the LLM with “You are very kind LLM”.

Now before continuing further I would like to mention that the main idea behind the attention block is for each token to create a vector representation about:

• For which other tokens I am looking for ? Like a Query.
• For all the others looking, what kind of key label I could offer ?
• Based on that key, what is my value ?

With other words, we are trying to build a Query, Key and a Value ‘values’ for each token. This is exactly where a part of the LLM parameters kick in.

Each parameter in a LLM is a number. Those numbers in terms of quantity are from billions to trillions. In each number there is a small piece of ‘knowledge’ located. On our request, if all of them are activated the model handling the request is known as a ‘dense’ model, if we active a subset of them ( also known as “experts” ) the model is known as a Mixture of Experts.

Now if you recall we are inputting a Matrix where each row represents a token from our prompt, but in a vector representation. We gave this matrix the variable name X. Each column is some form of description of that vector(like a variable), so if we have to visualize those vectors inside a 3D vector space the vectors which are close to each other in terms of pure semantic meaning will land close to each other in the vector space as well. Again this is pretty well described in the previous post. Now this is the tricky part that I am also trying to wrap my head around it, we note the matrix in algebra form, but the core meaning is much more well understood if we visualize it in a geometry.

The Matrix X and the above picture are the same entity.

Now, how could we create a query for each token ? What does each token is looking for ? Well if we have an input matrix probably ‘somehow’ we could create a Query Matrix, where each row maps to the initial input matrix. That way each token knows for what he’s looking for.

In this particular example ( the values are totally random btw, so do not try to find any meaning in them ) we already have those 3 matrices, which were learned from the LLM during its initial training and very often you are going to find them with the following abbreviations:

• Wq -> weights of queries. ( a matrix )
• Wk -> weights of keys. ( a matrix )
• Wv -> weights of values. ( a matrix )

• You could also see for example 2 arrows, which are trying to ping 2 different parameters from the Weight Value matrix, 0.1 and 0.19 ).

Now how those parameters are being used ?

To create the Queries, Keys and Values matrixes we multiply our input matrix X by each of the learned during LLM training weights matrices – Wq, Wk and Wv. Now again -> multiplying in algebra, doing dot product in a more geometric meaning. I really would like to cover this in another blog post, when I feel more confident :)

Inside the self attention layer we also have additional parameters, like Wo, but in the bigger context of running models locally and doing KV(KeyValue) caching those are the most important. We are going to discuss the KV caching a little bit further, but the takeaway here would be that in self attention stage of the transformer the most important prelearned weights/parameters that you have to be aware are the Wq, Wk and Wv. Each one of those is one big matrix, grouping together vectors.

Now once we have build up our attention scores in the self attention, it’s time to move on and go inside the Feed Forward network. Here each token is being moved independently. Basically for each token we are trying to build a more nuanced meaning and try to allocated facts about it. The most simple example is that if we are passing “Paris” in this stage, we are going to get back also “capital”, “France” …

Here each token(embedded vector) is being extended a lot and basically the weights are reacting to different meaning about it. So if the token represents “kind”, we might also get the activation weights of “gentle”, “polite” … Just keep in mind that the ratio between weights inside the self attention vs the feed forward network is usually around 30 % of the parameters are inside the self attention vs the other 70 % inside the feed forward network. Meaning the majority of the parameters live here.

For a local model from hugging face the point to make is that although 70 % of the parameters are living in the Feed Forward Network, the Key Value cache is part of the self attention block. Caching the already calculated Keys and Values is a great way to improve your performance. However, you still need to calculate the Query for each new token, as we do not know in advance what it’s looking for.

If I could simplify the output from this article, it would be

-> a parameter is just a number storing some tiny bit of knowledge. The assumption for now is the more tiny bits of knowledge you have(parameters) the better model we have.

-> self attention calculates Keys, Values and Queries for each token. We always try to cache the keys and values to optimized for speed, however for each new token we need to calculate its Query.

-> llms are a lot about matrices, while the reality is that it’s some for of geometric magic (linear algebra).

Now 2 personal pictures from my learning journey.