These LLMs begin with text… sometimes lots of it! It serves two purposes. First, it shows these models how to speak (how words generally fit together). Second, the text can also embed facts so models "know" things. As we will see, these networks might be closer to a search engine that is very flexible... it might not understand exactly the same way you understand.

In any case, let's start by putting in some text from which we can learn! This is called a corpus. We will have this model learn how to answer questions so put one question and answer per line like "Who discovered DNA? Watson and Crick" with a question mark in the middle. In practice, LLMs are generally trained off of lots of text but our model will be small so we will stick to the question and answer format.

With all of that in mind, either move ahead with the default corpus or modify the corpus in the textbox (like by adding a question and answer of your own)! When done, click the "Use Corpus" button. If you want to start over on your work, click "Reset Corpus" and it will go back to the default.

We have our text which will be used to shape the neural network. Next, let's actually make that system. At the root of all of this are neurons.

Similar to the biological pieces that build our brains, neurons in computers can "activate" or light up on different signals like when they see a specific word. To turn words into something the computer can understand, each word ("token") is given a unique ID. This mapping from token to number is called an encoding. In our neural network, we will have one neuron per ID which will activate when it sees its word.

Try typing a word which appeared in your corpus to see its corresponding neuron light up. Note that, if you type a word it hasn't seen before, no neurons light up.

Not sure what to type? Try "plants" though any word from your corpus will work. If you type multiple words, only the first one will be used here.

There are a series of layers where neurons from one layer "connect" or pass information on to neurons in the next layer. A neuron from one layer can encourage or discourage a neuron in the next layer from activating. We will focus on "dense" networks where all neurons in one layer connect to the next.

These connections are where the magic happens. Maybe there are two neurons in the first layer: "who" and "moon" (both of these are tokens). Maybe they both connect to the same neuron in the second layer. If both "who" and "moon" are present in a question (both the "who" and "moon" neurons light up), this second layer neuron lights up. This second layer neuron might represent the concept of Neil Armstrong.

This finally reveals the overall structure of these systems.

At the start is an "input" layer. This is where our neurons are activated by the "prompt" or, in this case, our question. From the previous example, the "who" and "moon" neurons would be in the input layer.

In the middle are "hidden" layers. These are places where we have room to store these associations. This can get a little complex. However, you can generally assume that our network can hold more information as we add more hidden layers.

At the end of the hidden layers is an "output" layer with one neuron per word that lights up with the network's prediction. To predict the full answer (which may contain multiple tokens), we run the network in repetition. It writes one word after another.

Why does this matter? Most LLMs predict what word comes next. So, given the input of "What is the capital of France?" it would predict "Paris" (hopefully). If given "Who discovered DNA? Watson and" it would predict "Crick" (hopefully). Our network learns both the structure of language and (some) facts.

All that in mind, let's make some room for our memories in these networks of connections. Let's specify the layers our network will have. You can play around with how many neurons are in each layer. Not sure what to choose? You can leave them at the defaults!

After you specify the sizes of these layers, there's one last detail we need to cover: context. This is a word you might hear thrown around when discussing LLMs. We will cover that in the next step.

Think back to the input layer. How many neurons are there? It's the number of known words multiplied by the longest piece of text a network can handle. This is the "context window" or "context size" and larger context means more words given to a network before it predicts the next one (which, again, it does in repetition until it thinks it has reached the end of the answer).

Try typing just a few words and see how neurons in the input layer light up. Here, we only visualize the first three tokens (words) you write so we only are seeing the first part of the input. All of the circles you see here are in the input layer. This defines the context.

Not sure what to try? If you used the example corpus, type "France and Germany" and see what happens! Here, we turn on three of our many hundreds of neurons in the input layer to indicate "France" was in the first position, "and" was in the second position, and "Germany" was in the third position. If you are paying close attention, you'll notice that each question or "prompt" corresponds to a unique combination of activated neurons.

You have everything you need to build a neural network. However, while we have the "architecture" of the model, it hasn't actually learned anything yet. Next, we will do a "training" where the neural network reads the corpus over and over to "learn" associations.

The network starts off with random connections between neurons. It makes a guess to the answer (next word) and, if it's right, the neural pathways that fired to give that prediction are strengthened so it's more likely to get the right answer the next time. If it is wrong, those neural pathways are weakened. The result are "weights" which define how neurons activate or inhibit each other given the corpus. This is called backpropagation.

Alright, finally, let's train the network.

Here you can specify how intensely to train. Most users can leave this at medium. However, you can reduce this to low training intensity for a poorer performing model but one that is built quickly (good for old machines). Alternatively, go for high training intensity if you have a powerful computer and some time. Here, the number of "epochs" refers to the number of times it reads all of the corpus. Note this might take a moment.

We've finished training, now we can use our weights. This step is called inference and tends to be quite a bit faster than training. Go ahead and try asking it a question that appeared in your corpus and a question that didn't.

For example, try "When did the Berlin Wall fall?" first. Then, maybe try "Where is Mexico?" afterwards. These appeared in your corpus (unless you removed them). Also, try asking "Where is Berlin?" or similar (which didn't appear in the corpus unless you added them).

You likely will find that the model will confidently give an incorrect answer. This is called a hallucination though one could also just call these errors. This reveals the crucial insight that predicting the next word can encode knowledge but these are all just words to our network. It might not accurately say when it's unsure about its answer.

Note: It is recommended you only ask one question at a time.

How does this compare to the big models out there? There are proprietary models you have to pay the developer for. There are open models that are free for everyone. Some are big models you probably can't run on your own machine. However, some also run on your computer or phone without an internet connection.

Some LLMs you would use train from a massive corpus of millions of books. This means that, whereas the model we just built has about a million connections or "parameters" as they are called, some LLMs may have tens to hundreds of billions (for example, 300B means 300 billion parameters). These may not all be dense either. Furthermore, some LLMs may use tools like a web search engine instead of giving you an answer from their memory. This may help cut down on errors. In this case, they may generate search queries to run and write text (like a short answer) summarizing another piece of text (like a full web page).

Even so, you can start to see the potential for "AI" and their limits. Keep in mind that things which can listen to and respond in words don't necessarily know what the words mean (at least not quite the same way we do). Some people call them parrots for that reason. They may just know what word is likely to come next.