Fine-Tuning
Large Language Models

Compiled by Mayank Chugh

SFT PEFT LoRA QLoRA RLHF DPO Preference Alignment
01 · Definition

What is fine-tuning?

Fine-tuning is the process of taking an already-trained model and training it further on domain-specific or task-specific data so that it performs better for a particular use case.

"Fine-tuning is a process of taking an already-trained model and training it further on your domain-specific data for better performance." — Course notes

The key insight is that training a large language model from scratch is extremely difficult (massive compute, massive data, massive time). Instead, we leverage the general knowledge already embedded in a publicly-available open-source model, and specialise it for our needs.

Practical analogy: rather than teaching someone everything from kindergarten, you hire a knowledgeable professional and give them a short domain-specific onboarding course.
02 · Overview

The LLM training pipeline

Every production LLM goes through (at most) three sequential training stages. Understanding these is the foundation for understanding fine-tuning.

1
Pre-training (raw model)
e.g. meta-llama/Meta-Llama-3-8B
Trained on internet-scale raw text data (unsupervised / self-supervised learning). The model learns to predict the next token. It acquires broad world knowledge but has no idea how to follow instructions. This is the base or "raw" model — it just knows language.
2
Supervised Fine-Tuning / Instruction Tuning (SFT)
e.g. meta-llama/Meta-Llama-3-8B-Instruct
Trained on paired input → output data so the model learns to follow instructions. Meta collected instruction datasets, presented them as (input, output) pairs, and fine-tuned the raw model. This produces the "Instruct" variant. The model now knows how to have a conversation, answer questions, and follow user prompts.
3
Preference Alignment (optional but increasingly common)
e.g. OpenRLHF/Llama-3-8b-rlhf-100k
The model is further trained to respond according to human preferences. Human annotators label which of two model responses they prefer. That preference data is used to retrain the model. Not mandatory — but large companies (OpenAI, Anthropic, Google) all do this now. Introduced prominently by ChatGPT / OpenAI.
03 · Model stages

The three stages of an LLM (with Llama example)

Using Meta's Llama 3 8B as a concrete example — the same model family appears at each stage, and you can download any of them from Hugging Face:

Stage HuggingFace model ID What it knows
Pre-trained (raw) meta-llama/Meta-Llama-3-8B Next token prediction, broad world knowledge, no instruction-following
Instruction-tuned (SFT) meta-llama/Meta-Llama-3-8B-Instruct How to follow instructions, conversation, Q&A format
Preference-aligned (RLHF) OpenRLHF/Llama-3-8b-rlhf-100k How to respond in a way humans prefer; aligned to human values
You can pick the model from any of these stages and apply your own custom fine-tuning on top of it. The right choice depends on your use case (see Section 4).
04 · Strategy

Choosing the right base model for custom fine-tuning

The choice of starting stage depends entirely on what output you want from your fine-tuned model. A useful example is a pharma company use case:

Use case: A pharma company has internal documents on molecule studies and wants an AI model to work with that domain-specific data.
📄
Pure text / data generation
Goal: generate new pharma text (not Q&A). Start from the raw pre-trained model (Meta-Llama-3-8B) and fine-tune on your raw domain documents.
💬
Chatbot / Q&A / conversational AI
Goal: question-answering system over pharma data. Start from the instruction-tuned model (Meta-Llama-3-8B-Instruct) and fine-tune on your domain-specific (input, output) pairs.

The key point: the instruction-tuned model already knows how to follow instructions in general. Your custom SFT then teaches it pharma-specific instructions. This is almost always the better starting point for chatbot use cases.

Training an LLM from scratch is extremely difficult and expensive. Always start from an open-source base model and fine-tune it. Companies like Meta, Mistral, Alibaba, and Google make high-quality base models freely available.
05 · SFT

Supervised fine-tuning (SFT)

What is SFT?

SFT = Supervised Fine-Tuning. Also called Instruction Fine-Tuning (IFT). In supervised learning we have labelled data: for every sample, we have an input and a corresponding output.

In the context of LLMs, the SFT dataset looks like this:

Input column A question, instruction, or prompt (the user turn) Output column The desired model response (the assistant turn)

Search "instruction fine-tuning dataset" on Hugging Face to see hundreds of examples.

What does SFT teach the model?

It teaches the model how to follow instructions: given a user prompt, how should the model generate a relevant, well-formed response. Before SFT, the pre-trained model only knows next-token prediction — it does not know the concept of "user" or "assistant."

Libraries used

Hugging Face Transformers Unsloth

Both will be demonstrated in upcoming practical sessions. Unsloth is particularly useful for efficient fine-tuning with limited compute.

06 · Internals

Parameter-level understanding

To understand fine-tuning deeply, you need to understand what is actually being updated inside the model.

What is a parameter?

A parameter = weights and biases of the neural network. In a transformer-based LLM, parameters exist in two main places:

Self-attention layer
Contains the Query (Q), Key (K), and Value (V) weight matrices. These are learned during training.
Feed-forward neural network (FFNN)
Contains its own weights and biases. Also trained during the learning process.

When we "fine-tune" a model, we are retraining some or all of these weights on new data. The question is: which ones?

07 · Approaches

Full fine-tuning vs partial fine-tuning

Type What gets updated Cost Practical?
Full fine-tuning All parameters (weights & biases) Huge GPU power + massive infrastructure No — we will NOT do this
Partial fine-tuning A subset of parameters Manageable; can run on a single GPU Yes — this is our focus
Full fine-tuning requires enormous GPU infrastructure (think: hundreds of A100s). It is what companies like Meta do when building the base model. For custom/domain fine-tuning, we always use partial fine-tuning.
08 · Old-school method

Old-school partial fine-tuning (layer freezing)

Before PEFT, the standard technique for partial fine-tuning was layer freezing. This was used extensively with CNNs (VGG, ResNet) and early LLMs (BERT, BART, T5).

Method 1 Freeze all layers → train only last output layer
All existing weights are frozen (not updated). Only the final output layer's weights are retrained. Effective for simple task adaptation but very limited.
Method 2 Freeze early layers → retrain remaining last layers
The first N layers (which learn generic, low-level representations) are frozen. The later layers (which encode more task-specific knowledge) are retrained. More powerful than Method 1.

Why this fails for modern LLMs

Modern LLMs (Llama, Mistral, GPT-4 class) have enormous architectures — billions of parameters, dozens or hundreds of layers. Even "partial" retraining of the last few layers requires massive infrastructure. The old-school method does not scale.

The old-school method works fine for CNN architectures and smaller LLMs like BERT. For billion-parameter models, we need a completely different approach: PEFT.
09 · Modern approach

PEFT — Parameter-Efficient Fine-Tuning

PEFT = Parameter Efficient Fine-Tuning. It is the umbrella term for a family of modern techniques that allow fine-tuning large models using only a small number of additional or selected parameters.

Full form Parameter Efficient Fine-Tuning Key idea Train a tiny subset of parameters (or add a tiny set of new ones), not the whole model Infrastructure needed Single GPU + small VRAM (VRAM = RAM inside the GPU) Use in course This is where we spend most of our time in the SFT chapter
"PEFT is just a parent term. LoRA is a PEFT technique — like how 'man' and 'woman' are both human." — Course notes

PEFT techniques covered in this course

LoRA Low-Rank Adaptation — the industry standard. Attaches a small "adapter" to the model. Main focus of this course.
QLoRA Quantised LoRA — LoRA applied to a quantised (compressed) model. Reduces memory further.
DoRA Weight Decomposition Low-Rank Adaptation — a variant of LoRA that applies weight decomposition. More optimised, latest technique. Given as an assignment.
BitFit Only trains the bias terms. Extremely lightweight but limited. Not used in this course.
IA³ Based on attention mechanism modification. Not used in this course.
10 · LoRA family

LoRA, QLoRA, and DoRA explained

LoRA — Low-Rank Adaptation

Instead of retraining the original weight matrices W, LoRA adds a small adapter alongside them. The adapter consists of two small low-rank matrices (A and B) whose product approximates the weight update: ΔW = A × B.

Core idea Adapter injection
The original weights W are frozen. Only the adapter matrices A and B are trained. After training, the adapter is merged back: W_new = W + AB. This is far more memory-efficient than training W directly.
LoRA is the most widely used PEFT technique in the industry. If you know only one fine-tuning technique, know LoRA.

QLoRA — Quantised LoRA

Quantisation = reducing numerical precision to save memory. Instead of storing weights as 32-bit or 16-bit floats, we convert them to smaller representations:

float32 → float16 / bfloat16 → int8 / int4

QLoRA = load the model in quantised form (saving VRAM), then apply LoRA adapters on top. You get almost the same quality as LoRA but with dramatically lower memory requirements.

DoRA — Weight Decomposition LoRA

DoRA decomposes the pretrained weight matrix into a magnitude component and a direction component, then applies LoRA-style updates to the direction. It is more expressive than standard LoRA and is the latest state-of-the-art PEFT technique.

TechniqueMemoryExpressivenessIn course?
LoRALowHighMain focus
QLoRAVery lowHighCovered
DoRALowVery highAssignment
BitFitMinimalLowNot covered
IA³MinimalLowNot covered
11 · Stage 3

Preference alignment

After SFT, a model knows how to follow instructions — but it may not respond the way humans prefer. Preference alignment is Stage 3 of the pipeline: training the model to match human preference.

How is preference data collected?

In typical preference-collection flows (similar to ChatGPT-style interfaces), users are shown two responses to the same prompt and asked "Which response do you prefer?" This generates preference-annotated data — each sample records which response was chosen and which was rejected.

The data format for preference tuning includes: a prompt, a "chosen" response (preferred by humans), and a "rejected" response (not preferred). This is sometimes called a triplet: (prompt, chosen, rejected).

Is it mandatory?

Preference alignment is not mandatory every time, but it is now standard practice at large AI companies. It makes models safer, more helpful, and more aligned with human values. OpenAI introduced it with ChatGPT; it has since been adopted by Anthropic, Google, Meta, and others.

12 · Algorithms

Preference alignment algorithms

1
RLHF — Reinforcement Learning from Human Feedback
Used by: OpenAI (ChatGPT). Algorithm behind it: PPO
RLHF is a framework, not a single algorithm. It uses reinforcement learning — specifically the Deep RL branch — and the underlying optimisation algorithm is PPO (Proximal Policy Optimisation). The model is treated as a "policy" that is updated to maximise a reward signal derived from human preference judgements.
2
GRPO — Group Relative Policy Optimisation
Used by: DeepSeek (Chinese company)
An updated variant of RLHF / PPO. Used to train DeepSeek models. Understanding RLHF first will make GRPO straightforward. Assigned as independent study after RLHF is covered.
3
DPO — Direct Preference Optimisation
Used by: many modern LLMs. No RL required.
DPO removes the need for a separate reward model and reinforcement learning. It directly optimises the model on (chosen, rejected) pairs using a cross-entropy-style objective. Simpler and often more stable than RLHF. Will be covered in depth in this course.
4
ORPO — Odds Ratio Preference Optimisation
Updated version of DPO
The latest update to DPO. More efficient and effective. Understanding DPO will make ORPO easy. Assigned after DPO is covered.

What will be taught in this course

RLHF (with PPO) — taught in depth DPO — taught in depth GRPO — assignment (self-study) ORPO — assignment (self-study)
13 · Tooling

Tools: Hugging Face & Unsloth

Hugging Face Transformers / PEFT / TRL
The primary ecosystem for fine-tuning. The transformers library provides model loading; peft provides LoRA/QLoRA; trl (Transformer Reinforcement Learning) provides SFT trainers, DPO trainers, and RLHF utilities. All practical sessions will use these.
Unsloth
A newer, highly optimised fine-tuning library that makes LoRA and QLoRA 2–5× faster with lower VRAM usage. Works on top of Hugging Face. Excellent for free-tier Google Colab. Will be demonstrated in upcoming sessions.

Hardware note

All fine-tuning assignments will be done via Google Colab — you do not need a local GPU. If you have at least 16 GB of RAM locally, you can experiment with Ollama-based models, but training/fine-tuning should be done on Colab.

14 · Revision

Complete summary & mental model

The hierarchy to remember

FINE-TUNING HIERARCHY

Custom Fine-Tuning = train your model on your own data
SFT (Instruction Tuning) input → output pairs
Full Fine-Tuning ← Not practical (huge infra)
Partial Fine-Tuning (PEFT) ← Our focus
Old-school: layer freezing (CNN, BERT, T5 era)
Modern PEFT: LoRA QLoRA DoRA BitFit IA³
Preference Alignment chosen / rejected pairs
RLHF + PPO (OpenAI)  ·  GRPO (DeepSeek)  ·  DPO  ·  ORPO

Five sentences to remember

  1. Fine-tuning = taking a pretrained model and retraining it further on your own domain data.
  2. Every LLM exists at three stages: raw (pre-trained), instruction-tuned (SFT), preference-aligned (RLHF/DPO).
  3. You can start from any stage and apply custom fine-tuning on top — choose based on your use case.
  4. Full fine-tuning is too expensive; we always use partial fine-tuning (PEFT), with LoRA as the main technique.
  5. Preference alignment (RLHF, DPO) teaches the model to respond the way humans prefer, using (chosen, rejected) response pairs.
Next session: deep dive into practical SFT with Hugging Face & Unsloth. Google Colab will be set up. The pharma use case will be used as the hands-on example throughout.