Demystifying LLMs

Devendra Singh Chaplot

Mistral AI

Feb 13, 2024

Mistral AI
Co-Founders

Arthur Mensch Timothée Lacroix Guillaume Lample

CEO CTO Chief Scientist
Former AI researcher at Former AI researcher at Former AI Researcher at
DeepMind, Polytechnique alum Meta, ENS alum Meta, Polytechnique alum

Releases

$500M+ funding, Of ces in Paris/London/SF Bay Area

fi
Mistral AI LLMs
Contents
• Stages of LLM Training:

• Pretraining

• Instruction-Tuning

• Learning from Human Preferences: DPO/RLHF

• Evaluation of LLMs

• Retrieval Augmented Generation (RAG)

• Recipe for RAG with code

Stages of LLM Training
1. Pretraining

2. Instruction-Tuning

3. Learning from Human Feedback

Stages of LLM Training
1. Pretraining

2. Instruction-Tuning

3. Learning from Human Feedback

Pretraining
Pretraining
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same
architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e.
experts). For every token, at each layer, a router network selects two experts to process the current
state and combine their outputs. Even though each token only sees two experts, the selected experts
can be different at each timestep. As a result, each token has access to 47B parameters, but only uses
13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it
outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular,
Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks.
We also provide a model
fi
Pretraining
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same
architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e.
experts). For every token, at each layer, a router network selects two experts to process the current
state and combine their outputs. Even though each token only sees two experts, the selected experts
can be different at each timestep. As a result, each token has access to 47B parameters, but only uses
13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it
outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular,
Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks.
We also provide a model

Large Language Model

O(1-100B) parameters

We
fi
Pretraining
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same
architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e.
experts). For every token, at each layer, a router network selects two experts to process the current
state and combine their outputs. Even though each token only sees two experts, the selected experts
can be different at each timestep. As a result, each token has access to 47B parameters, but only uses
13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it
outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular,
Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks.
We also provide a model

introduce

Large Language Model

O(1-100B) parameters

We
fi
Pretraining
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same
architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e.
experts). For every token, at each layer, a router network selects two experts to process the current
state and combine their outputs. Even though each token only sees two experts, the selected experts
can be different at each timestep. As a result, each token has access to 47B parameters, but only uses
13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it
outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular,
Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks.
We also provide a model

introduce Mixtral

Large Language Model

O(1-100B) parameters

We introduce
fi
Pretraining
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same
architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e.
experts). For every token, at each layer, a router network selects two experts to process the current
state and combine their outputs. Even though each token only sees two experts, the selected experts
can be different at each timestep. As a result, each token has access to 47B parameters, but only uses
13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it
outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular,
Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks.
We also provide a model

introduce Mixtral 8x7B , a Sparse Mixture

Large Language Model

O(1-100B) parameters

We introduce Mixtral 8x7B , a Sparse

fi
Pretraining
• Task: Next token prediction
Pretraining
• 1 token ~= 0.75 word
introduce Mixtral 8x7B
• Vocab size: O(10K) tokens
• Each token is represented by an integer
Large Language Model
(LLM)

We introduce Mixtral
Pretraining
• Why is it hard?
Pretraining
• Why is it hard?
• Time: Datasets are huge - O(1T) tokens
• Preprocessing, Cleaning, Deduplication
• More data might not lead to better model

Llama pretraining data mixture

Pretraining
• Why is it hard?
• Time: Datasets are huge - O(1T) tokens
• Preprocessing, Cleaning, Deduplication
• More data might not lead to better model
• Money: O(1-100B) parameters
• O(1-10K) GPUs for weeks or months Llama pretraining data mixture
• O($10-100M) per model
Pretraining
• Why is it hard?
• Time: Datasets are huge - O(1T) tokens
• Preprocessing, Cleaning, Deduplication
• More data might not lead to better model
• Money: O(1-100B) parameters
• O(1-10K) GPUs for weeks or months Llama pretraining data mixture
• O($10-100M) per model
• YOLO: decide model architecture, hyper
parameters, data mixture for the “big run”
• Best hyper-parameters for a smaller model
Llama Model sizes, architectures, and
might not be the best for a larger model optimization hyper-parameters.
Can we use the Pretrained model?
Can we use the Pretrained model?
Prompt:

Write a python function to nd whether the input number is prime.

fi
Can we use the Pretrained model?
Prompt:

Write a python function to nd whether the input number is prime.

Response:
Example:

Input: 17
Output: True

Input: 15
Output: False

Approach:

1. Let’s assume that the input number is n

2. Check if n is divisible by 1. If n is divisible by 1 then it is not a prime number.
3. Check if n is divisible by 2. If n is divisible by 2 then it is not a prime number.
fi
Can we use the Pretrained model?
Prompt:

def is_prime(x: int):

"""
takes as input an integer x. Returns True if x is prime and False otherwise
"""
Can we use the Pretrained model?
Prompt:

def is_prime(x: int):

"""
takes as input an integer x. Returns True if x is prime and False otherwise
"""

Response:

if x <= 1:
return False
for i in range(2, int(x ** 0.5) + 1):
if x % i == 0:
return False
return True
Can we use the Pretrained model?
Prompt:

def is_prime(x: int):

"""
takes as input an integer x. Returns True if x is prime and False otherwise
"""

Response:

if x <= 1:
return False
for i in range(2, int(x ** 0.5) + 1):
if x % i == 0:
return False
return True

Model knows the answer but it is not aligned with human preferences
Stages of LLM Training
1. Pretraining

2. Instruction-Tuning

3. Learning from Human Feedback

Instruction Tuning
Prompt: [INST] Write a python function to nd whether the input number is prime. [\INST]

def is_prime(x: int):

“""
takes as input an integer x. Returns True if x is prime and False otherwise
"""
if x <= 1:
Response: return False
for i in range(2, int(x ** 0.5) + 1):
if x % i == 0:
return False
return True
fi
Instruction Tuning
Prompt: [INST] Write a python function to nd whether the input number is prime. [\INST]

def is_prime(x: int):

“""
takes as input an integer x. Returns True if x is prime and False otherwise
"""
if x <= 1:
Response: return False
for i in range(2, int(x ** 0.5) + 1):
if x % i == 0:
return False
return True

Large Language Model

O(1-100B) parameters

[INST] Write … [\INST]

fi
Instruction Tuning
Prompt: [INST] Write a python function to nd whether the input number is prime. [\INST]

def is_prime(x: int):

“""
takes as input an integer x. Returns True if x is prime and False otherwise
"""
if x <= 1:
Response: return False
for i in range(2, int(x ** 0.5) + 1):
if x % i == 0:
return False
return True

def

Large Language Model

O(1-100B) parameters

[INST] Write … [\INST] def

fi
Instruction Tuning
Prompt: [INST] Write a python function to nd whether the input number is prime. [\INST]

def is_prime(x: int):

“""
takes as input an integer x. Returns True if x is prime and False otherwise
"""
if x <= 1:
Response: return False
for i in range(2, int(x ** 0.5) + 1):
if x % i == 0:
return False
return True

def is_prime

Large Language Model

O(1-100B) parameters

[INST] Write … [\INST] def

fi
Instruction Tuning
Prompt: [INST] Write a python function to nd whether the input number is prime. [\INST]

def is_prime(x: int):

“""
takes as input an integer x. Returns True if x is prime and False otherwise
"""
if x <= 1:
Response: return False
for i in range(2, int(x ** 0.5) + 1):
if x % i == 0:
return False
return True

def is_prime (x) :

Large Language Model

O(1-100B) parameters

[INST] Write … [\INST] def is_prime (x)

fi
Instruction Fine-tuning

• Dataset Instruction-tuning

• Paired: (Prompt, Response)

def is_prime
• O(10-100K instructions)

Large Language Model

(LLM)

[INST] … [\INST] Def

Instruction Fine-tuning

• Dataset Instruction-tuning

• Paired: (Prompt, Response)

def is_prime
• O(10-100K instructions)
• Task:
Large Language Model
• Next word prediction (Masked) (LLM)

[INST] … [\INST] Def

Instruction Fine-tuning

• Dataset Instruction-tuning

• Paired: (Prompt, Response)

def is_prime
• O(10-100K instructions)
• Task:
Large Language Model
• Next word prediction (Masked) (LLM)
• Compute:
• O(1-100) GPUs [INST] … [\INST] Def

• Few hrs/days
Stages of LLM Training
1. Pretraining

2. Instruction-Tuning

3. Learning from Human Feedback

Human Preferences
Human preferences are cheaper/easier than human annotation

Prompt: [INST] Write a python function to nd whether the input number is prime. [\INST]

def is_prime(x: int):

“""
takes as input an integer x. Returns True if x is prime and False otherwise
"""
if x <= 1:
Response 1: return False
for i in range(2, int(x ** 0.5) + 1):
if x % i == 0:
return False
return True
def is_prime(x: int):
“""
takes as input an integer x. Returns True if x is prime and False otherwise
"""
if x <= 1:
Response 2: return False
for i in range(2, x):
if x % i == 0:
return False
return True
fi
Human Preferences
Human preferences are cheaper/easier than human annotation

Prompt: [INST] Write a python function to nd whether the input number is prime. [\INST]

def is_prime(x: int):

“""
takes as input an integer x. Returns True if x is prime and False otherwise
"""
if x <= 1:
Response 1: return False
for i in range(2, int(x ** 0.5) + 1):
if x % i == 0:
return False
return True
def is_prime(x: int):
“""
takes as input an integer x. Returns True if x is prime and False otherwise
"""
if x <= 1:
Response 2: return False
for i in range(2, x):
if x % i == 0:
return False
return True
Response 1 > Response 2
fi
Reinforcement Learning
from Human Feedback (RLHF)

[Deep Reinforcement Learning from Human Preferences. Christiano et al. 2017]

Direct Preference Optimization (DPO)

[Deep Reinforcement Learning from Human Preferences. Christiano et al. 2017]

[Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Rafailov et al. 2023]
Stages of LLM Training
Pretraining Instruction-Tuning Learning from Human Feedback

Dataset: Dataset: Dataset:

Raw text Paired: (Prompt, Response) Human Preference Data
Few trillions of tokens O(10-100K instructions) O(10-100K)
Task: Task: Task:
Next word prediction Next word prediction (Masked) RLHF/DPO
Compute: Compute: Compute:
O(1-10K) GPUs O(1-100) GPUs O(1-100) GPUs
Weeks/months of training Few hrs/days Few hrs/days
Evaluation of LLMs
Evaluation of pretrained models
Evaluation of pretrained models
0-shot:

def is_prime(x: int):

"""
takes as input an integer x.
Returns True if x is prime and False otherwise
"""
Evaluation of pretrained models
0-shot:

def is_prime(x: int):

"""
takes as input an integer x.
Returns True if x is prime and False otherwise
"""

3-shot:
## How old is Barack Obama in 2014?

Barack Obama is 57 years old in 2014.

## What is Barack Obama’s birthday?

Barack Obama was born on August 4, 1961.

## What is the name of Barack Obama’s wife?

Barack Obama’s wife is Michelle Obama.

## How tall is Barack Obama?

Evaluation of pretrained models
0-shot:

def is_prime(x: int):

"""
takes as input an integer x.
Returns True if x is prime and False otherwise
"""

3-shot:
## How old is Barack Obama in 2014?

Barack Obama is 57 years old in 2014.

## What is Barack Obama’s birthday?

Barack Obama was born on August 4, 1961.

## What is the name of Barack Obama’s wife?

Barack Obama’s wife is Michelle Obama.

## How tall is Barack Obama?

Evaluation of Instruction-tuned models

https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard
Evaluation of Instruction-tuned models

LMSYS Chatbot Arena Leaderboard

https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard
Evaluation of Instruction-tuned models
• Proxies for human evaluation:
• MT Bench:

• Ask GPT-4 to score responses

• 0.90 correlation with human
preferences
• Alpaca Eval:

• Compare win-rate against GPT-4 (v2)

• 0.84 correlation with human
preferences
Practical tips
• Proprietary vs Open-Source
• For proprietary models:
• Prompt Engineering: Few-shot prompting, Chain-of-thought
• Retrieval Augmented Generation (RAG)
Practical tips
• Proprietary vs Open-Source
• For proprietary models:
• Prompt Engineering: Few-shot prompting, Chain-of-thought
• Retrieval Augmented Generation (RAG)
• For open-source
• Everything above
• Task-speci c ne-tuning and DPO: Need data and a bit of compute
fi
fi
 Practical tips
Open-source Proprietary
• Proprietary vs Open-Source
• For proprietary models:
• Prompt Engineering: Few-shot prompting, Chain-of-thought
• Retrieval Augmented Generation (RAG)
• For open-source
• Everything above
• Task-speci c ne-tuning and DPO: Need data and bit of compute
• Balance performance vs cost (training and inference)
• Proprietary models higher general-purpose performance
• Open-source models can beat proprietary models on speci c tasks
with ne-tuning
• Proprietary models typically have higher inference cost
Price
0.42€ 1.8€ 7.5€
(per M tokens)
fi
fi
fi
fi
Retrieval Augmented Generation (RAG)
Retrieval Augmented Generation (RAG)
When do we need Retrieval Augmented Generation (RAG)?
• LLM doesn’t know everything, sometimes require task-speci c knowledge
• Sometimes you want LLMs to answer queries based on some data source to reduce hallucinations
• Knowledge resource doesn’t t in the context window of the LLM

[Figure from https://lemaoliu.github.io/retrieval-generation-tutorial/]

fi
fi
Recipe for RAG

[Figure from https://gradient ow.substack.com/p/best-practices-in-retrieval-augmented]

fl
Basic RAG code
https://docs.mistral.ai/guides/basic-RAG/