Data and AI Training

Home | Power BI | Excel | Python | SQL | Generative AI | Visualising Data | Analysing Data

How to build a Generative AI Model

This is in three stages

1. Build the base model
2. Supervised Fine Tuning
3. Reinforcement Learning

First step: self-supervised learning “guess the next word”

Download a lot of text (a corpus), preferably the entire internet.

Go through the next steps repeatedly for over the whole corpus in a process called self-supervised learning. This will cost $100m, takes several months and generate several hundred tons of CO₂.

• Remove the last part of a sentence.
• Let the LLM guess / predict the continuations - the next word(s).
• Adjust the model to match the actual continuations (the ground truth). After a while the model gets really good at this “guess the next word” game.

Next step: Reinforcement learning with human feedback (fine-tuning)

At this point the model is pre-trained and will just continue / complete text. Now we fine-tune the model to make it useful. Humans provide questions and finetune the model so that it responds with something close to a model answer. Another technique to fine tune is for the LLM to generate two different answers. People then indicate which response they prefer and this is fed back into the fine training.

As the model grows in size, it shows surprising “emergent behaviours”:

• be able to crack jokes,
• provide step-by-step instructions,
• perform chain of thought reasoning,
• act in a role,
• write poetry,
• solve algebraic equations and
• provide strategic advice to directors of large organisations!