Separate the training objective from the parameter-update method, then evaluate the adapted model against a clear task standard.
A language model can continue a sentence, follow an instruction, adopt a particular response format, and rank competing answers. Those behaviors do not all come from one training step. They reflect different data, objectives, and choices about which parameters are updated.
That distinction is useful whenever a product says its model has been “fine-tuned.” Fine-tuned on what? With which objective? Were all the parameters changed, or only a small adapter? Did the evaluation check accuracy, formatting, preference, or something else?
This guide explains pretraining, supervised fine-tuning, LoRA, QLoRA, reinforcement learning from human feedback, and Direct Preference Optimization. It includes three diagrams, worked parameter and memory calculations, and practical comparisons. The numbers in the examples are illustrative calculations, not benchmarks from a training run.
A map of the training stages

The diagram is a common route, not a universal recipe. Some models use additional objectives, continued pretraining, or other forms of reinforcement learning. A model can also be useful without passing through every box.
The distinction to keep is between an objective and an update method. Supervised fine-tuning describes learning from desired outputs. LoRA describes a way to restrict the trainable parameter update. You can use LoRA while performing supervised fine-tuning, and adapter methods can also support other objectives. The words describe different parts of the setup.
Pretraining: learning from token sequences
For a causal language model, a common pretraining objective is next-token prediction. A sequence supplies examples of which token followed the preceding context. The model produces scores, a loss compares them with the target tokens, and optimization changes the parameters [1].
Suppose a teaching sequence is “The cat sat on the mat.” Ignoring the details of a real tokenizer, earlier positions supply context for predicting later tokens. A model that gives the observed target a higher probability incurs a lower next-token negative log loss for that prediction.
Here is an original arithmetic example using natural logarithms:
Loss for an observed target token = −log(p_target)
p_target = 0.10 → loss ≈ 2.303
p_target = 0.50 → loss ≈ 0.693
p_target = 0.90 → loss ≈ 0.105The objective rewards probability assigned to the observed target. It does not contain a separate label saying that every sentence in the dataset is true. The dataset can contain disagreement, mistakes, fiction, repeated material, and changing facts. Learning to model the text distribution and verifying a claim are different tasks.
This helps explain why fluency is an incomplete evaluation. A response can resemble a well-written explanation while getting an important number wrong. Check the behavior you need rather than treating smooth prose as a direct measure of truth.
A base model and an instruction-following model
A base model has learned its pretraining objective. Its ability to continue text is not the same as consistently following a user's requested format, handling missing information well, or behaving like a useful assistant.
Supervised fine-tuning, often abbreviated SFT, trains on examples of the behavior you want. For conversational training, these can contain a prompt and a desired response. Training can select which portions of the sequence contribute to the loss, such as the response tokens, depending on the setup [2].
Imagine an illustrative dataset with this example:
Prompt:
Summarize this support request in one sentence.
Desired response:
The customer reports a duplicate charge and asks for a refund review.The example teaches a response pattern. It becomes more useful when the dataset includes varied requests, realistic ambiguity, and cases where the input lacks enough evidence. Hundreds of near-duplicate examples may teach a narrow template rather than broad task competence.
For an Arabic support assistant, the desired examples should reflect the language, register, terminology, and task it will face. Translating an English template does not by itself establish that the result handles Arabic names, dates, tone, or mixed technical text well.
Fine-tuning is a targeted change, not a guarantee
Fine-tuning can make a model more consistent at a task or style. Its value needs to be demonstrated on examples held out from training. If you repeatedly tune against the same test cases, those cases no longer provide a clean final assessment.
Write down the acceptance rule before training. For structured extraction, test field accuracy and schema validity. For summarization, check whether the answer retains the important conditions. For a support workflow, inspect both correct escalation and unnecessary escalation.
Also test abilities that you did not intend to change. A model can become more obedient to a narrow format while losing useful flexibility elsewhere. Improvements on a target task and regressions on other tasks should appear in the same evaluation report.
Full fine-tuning: updating the whole model
In full fine-tuning, the pretrained model's parameters are updated for the new objective. That gives a broad parameter space for adaptation, but the training setup also needs memory for working values such as gradients, optimizer state, and activations. The LoRA paper motivates its alternative partly by the cost of updating and storing full adapted models [3].
Do not estimate training hardware from the size of the weight file alone. A model that fits for inference may not fit for a chosen training configuration. Sequence length, batch size, precision, optimizer, and memory-saving techniques can all affect the requirement.
A larger update space does not automatically produce a better result. If the data is small, noisy, or poorly matched to deployment, full fine-tuning can spend more resources learning the wrong behavior. Compare its actual result with a smaller adaptation and a reasonable baseline.
LoRA: learning a smaller parameter update
Low-Rank Adaptation, or LoRA, freezes base weights and learns an update represented by smaller matrices. For one weight matrix, a common notation is:
Effective weight = W₀ + ΔW
ΔW = (α/r) BA
W₀: frozen base matrix
A and B: trainable adapter matrices
r: chosen rank
α: scaling parameterThe original paper introduces the low-rank update idea [3]. The PEFT documentation explains how it is applied and configured in practice [4]. Rank is a capacity choice for the update, not the number of labels or the number of layers in the base model.

Take a square matrix with 1,024 input features and 1,024 output features. The matrix alone has 1,048,576 parameters. For rank eight, choose A with shape 8 × 1,024 and B with shape 1,024 × 8:
Full matrix: 1024 × 1024 = 1,048,576 parameters
LoRA A: 8 × 1024 = 8,192
LoRA B: 1024 × 8 = 8,192
Adapter total = 16,384
Adapter/full-matrix fraction = 1.5625%
Full matrix / adapter = 64This is a parameter-count calculation for one chosen matrix, excluding biases and other modules. It does not establish a 64-fold reduction in total training memory or cost. The base model still exists, and the runtime still needs activations and other working memory.
LoRA adapters can make it practical to store several task adaptations without storing a complete independently fine-tuned model for each one. You still need the compatible base model and the correct adapter configuration. An adapter file is not a standalone language model.
QLoRA: quantized base, trainable adapters
QLoRA combines a low-precision frozen base with trainable LoRA adapters. The paper describes four-bit NormalFloat, double quantization, and paged optimizers as parts of its approach [5]. The key point is that adapter training does not require treating all base weights as full-precision trainable parameters.
Storage precision and computation precision are different choices. PEFT's quantization guide illustrates four-bit storage configuration with a separate computation dtype [6]. “Four-bit model” is therefore too vague to describe every tensor or operation in the training process.
Consider an idealized model with seven billion stored weights:
Swipe or scroll sideways to see every column.
| Assumed weight storage | Arithmetic | Raw weight storage |
|---|---|---|
| 16 bits per weight | 7 billion × 2 bytes | 14 GB, about 13.04 GiB |
| 4 bits per weight | 7 billion × 0.5 bytes | 3.5 GB, about 3.26 GiB |
GB here means decimal billions of bytes; GiB means powers of 1,024. These calculations exclude quantization metadata, non-quantized tensors, adapters, activations, optimizer state, and runtime overhead. They are not a promise that a particular seven-billion-parameter model can train on a device with 3.5 GB of memory.
The practical question is whether the chosen quantization, adapter placement, sequence length, and batch setup meet your quality and resource constraints. Lower storage precision can save memory while changing numerical behavior. Evaluate the trained result rather than assuming the compression is free.
Human feedback: comparing responses
Desired-response examples are one kind of supervision. Preferences are another. Given a prompt and competing answers, a reviewer can indicate which response better meets a stated criterion.
For example, a reviewer might prefer an answer that says the evidence is insufficient over an answer that invents a confident explanation. The labeling rule should make that expectation explicit. “Choose the better answer” can conceal disagreement over accuracy, length, tone, helpfulness, and risk.
Preferences are measurements collected under a process. Inspect that process: who labeled the data, what rubric they used, how disagreement was handled, and whether the examples represent real use. More labels cannot fix a rubric that rewards the wrong behavior.
Classic RLHF: reward model, then policy optimization
One influential RLHF recipe first performs supervised fine-tuning, learns a reward model from ranked responses, and then optimizes the language model using reinforcement learning. The InstructGPT paper describes such a pipeline [7]. It is an example of RLHF, not a claim that every feedback-trained model follows an identical recipe.
The reward model estimates how desirable a response is under its learned criteria. The policy is the model generating responses. In a classic PPO-based setup, policy updates use the reward signal while constraining aspects of the update; PPO is an optimization method for policies [8].
The reward score is not a truth detector. A policy can learn to produce responses that receive high reward while exploiting weaknesses in the reward model. Evaluation should therefore use checks beyond the training reward, including direct inspections of factuality and task completion where relevant.
DPO: learning directly from chosen and rejected responses
Direct Preference Optimization uses preference pairs in a direct training objective, avoiding the separate learned reward model and PPO rollout loop of the classic approach. The DPO paper derives its objective from a reward-regularized preference setting [9].
The training data commonly includes a prompt, a chosen response, and a rejected response. DPO adjusts relative response probabilities with reference-model terms and a configured strength parameter. The TRL trainer documentation describes the practical dataset and configuration [10].

“Simpler pipeline” does not mean “automatically better model.” If the chosen answers contain subtle mistakes, the model can be pushed toward those mistakes. If preferences only cover one narrow behavior, an evaluation must check what happened outside that behavior.
Compare the methods on the right axis
This table compares roles, not benchmark winners. Some rows can be combined in one system because they answer different questions.
Swipe or scroll sideways to see every column.
| Method | What it primarily changes | Training signal or mechanism | A useful question |
|---|---|---|---|
| Pretraining | Broad learned representations and sequence behavior | A pretraining objective such as next-token prediction | What data and objective established the base? |
| SFT | Behavior on demonstrated tasks | Desired outputs | Are examples representative and correct? |
| Full fine-tuning | Base parameters | Depends on the selected objective | Does the extra update freedom improve evaluation? |
| LoRA | Selected low-rank updates | Adapter parameters trained under an objective | Which modules and rank were chosen? |
| QLoRA | Memory representation plus adapter updates | Quantized frozen base and trainable adapters | What is stored at low precision, and what memory remains? |
| Classic PPO-based RLHF | Policy behavior under learned reward | Ranked responses, reward model, and RL | Does high reward correspond to actual task quality? |
| DPO | Relative preferences between responses | Chosen/rejected pairs and a direct loss | Is the preference dataset a reliable guide? |
Should you train at all?
Sometimes the problem is missing information rather than an unsuitable learned response pattern. A system answering questions about a changing handbook needs the current handbook. Retrieval-augmented generation supplies retrieved material to a generation process [11]. Fine-tuning and retrieval can coexist, but they solve different parts of a workflow.
For an illustrative customer-support product, a practical sequence might be:
- Define the task and write an acceptance standard.
- Try a capable available model with clear instructions and examples.
- Supply relevant, current documents when answers depend on external knowledge.
- Inspect failures and identify a repeated behavior gap.
- Consider fine-tuning when representative training examples can address that gap.
- Compare the adapted result with the original baseline on held-out tasks.
These are engineering choices, not a mandatory recipe. If a model already meets your requirements, training may add work without adding useful value. If it repeatedly fails a well-defined format or task despite adequate evidence, adaptation may be worth evaluating.
Build an evaluation before building the training run
Hold out examples that represent the work you intend to do. Include ordinary cases, difficult cases, absent information, and realistic ambiguity. Avoid near-duplicate training and test examples. Keep a final assessment separate from the examples you repeatedly inspect during tuning.
For a bilingual assistant, test Arabic and English separately as well as mixed-language inputs. For extraction, record field-level errors. For technical explanations, verify terminology and numerical examples. For tools, inspect the saved result and task completion rather than only the final message.
Record the training setup too: base checkpoint, tokenizer, data version, objective, trainable modules, rank if relevant, precision, batch setup, sequence length, optimizer, and checkpoint-selection rule. A successful-looking output is much easier to investigate when its configuration is known.
When comparing full fine-tuning with LoRA or QLoRA, use the same task standard and account for resource differences. A lower loss from a larger training budget is not automatically the better operational choice. Measure quality, inference behavior, storage, and the cost of maintaining the result.
A learning path with a concrete destination
Start by calculating the three token losses in this guide. Then distinguish a desired-response example from a preference pair. Next, reproduce the rank-eight parameter count and explain why it does not determine total memory. Finally, trace the two routes in the preference-training diagram.
The resources below separate original research from implementation guidance. The papers establish the methods and the scope of their experiments. Hugging Face documents training data formats, adapters, and quantization configuration. Read the material for the specific mechanism you want to understand rather than copying a large training recipe before its parts make sense.
My view for Saud Alotaibi’s Opinions is that “fine-tuned” should begin a technical explanation, not finish one. Readers deserve to know what was learned, which parameters changed, what evidence supports the improvement, and what the model still gets wrong. Those details turn a training label into something useful.
Sources & further reading
- [1] Hugging Face: training a causal language model
- [2] TRL: supervised fine-tuning trainer
- [3] Hu et al.: LoRA — Low-Rank Adaptation of Large Language Models
- [4] PEFT: LoRA conceptual guide
- [5] Dettmers et al.: QLoRA — Efficient Finetuning of Quantized LLMs
- [6] PEFT: quantization guide
- [7] Ouyang et al.: Training language models to follow instructions with human feedback
- [8] Schulman et al.: Proximal Policy Optimization Algorithms
- [9] Rafailov et al.: Direct Preference Optimization
- [10] TRL: DPO trainer
- [11] Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks