Fine-Tuning Small Language Models for Continual Learning On-Device with Self-Distillation

Abstract: Imagine a personal assistant that gradually learns how you write, which suggestions you accept, and how you use your apps—all without sending your private history to a training server. Recent model families, including Microsoft’s Phi-3, Meta’s quantized Llama 3.2 models, Google’s Gemma 3n, and Liquid AI’s LFM2, show that capable language models can now run directly on phones and other resource-constrained devices [1], [2], [3], and [4]. But local inference solves only half the problem:

How can the model continue learning from a single user when interactions are sparse and compute is limited to a phone?

In this post, we explore how to fine-tune a small language model (SLM) directly on a mobile device using on-policy self-distillation, building on Self-Distillation Enables Continual Learning [9]. The key idea is to use the current model as both student and teacher: the teacher receives additional context from an interaction and generates a better-informed training target, while the student learns to produce that output from the original input alone.

We adapt this idea to continual personalization from everyday phone interactions. The goal is to turn a sparse stream of user edits, choices, and outcomes into useful training signals while keeping both personal data and model updates on-device.

Three-step on-device SDFT loop: private activity arrives as a stream, an on-device language model acts, and a later hindsight observation loops back through self-distillation to teach the same model — Private activity arrives as a stream. The on-device language model acts first; after the phone observes what happened, SDFT uses that hindsight to teach the same model for future decisions.

Introduction

Why On-Device AI? A personal assistant becomes more useful when it can draw on private context such as messages, calendars, documents, and app activity. Proprietary language models accessed through remote APIs can process this information, but doing so often requires sending it off-device. An on-device small language model (SLM) can instead keep personal context, inference, and model updates on the user’s device. Recent model families—including Phi-3, quantized Llama 3.2, Gemma 3n, and Liquid AI’s LFM2—show that capable models can now run on phones and other edge devices [1], [2], [3], and [4]. This makes local inference increasingly practical and raises the next question: can the model also continue learning after deployment?

Why Continual Learning? A useful assistant should adapt to how each user writes, which suggestions they accept, and when they prefer to be interrupted. These behaviors differ across users and change over time, so a static model cannot remain personalized. For example, consider push-notification routing [5] and [6]. An on-device model receives a notification plus local context and chooses whether to deliver it now, save it for a later digest, or archive it. The phone may later record only what happened on the route it actually took—such as an immediate open, an open from the digest, a deletion, or no observable selection. That outcome becomes hindsight for the model’s future decisions.

Push-notification routing example on a device: an incoming project update enters an on-device language model; it can deliver now, save for later, or archive; save for later is selected and the phone later observes an open from the digest, while the unchosen route outcomes remain unknown — Push-notification routing is one example of action-dependent observation. The on-device language model chooses one route now and later observes only the factual outcome exposed by that route. That outcome becomes hindsight for future decisions; the unchosen outcomes remain unknown.

Continual learning on-device is hard because explicit user feedback is scarce. Proprietary models can be trained on large curated datasets or interaction logs from many users, allowing noisy signals to be averaged across large batches. An on-device SLM instead sees a sparse, asynchronous stream from one user, and most interactions do not reveal the correct action. An LLM memory system can store past interactions and retrieve similar cases as in-context examples, but each new decision depends on selecting and inserting the right memories [7]. Fine-tuning complements memory by making user-specific behavior part of the model itself, without requiring a matching example in the context every time [8]. The central challenge is therefore to turn sparse and delayed hindsight observations into useful supervision. On-policy self-distillation addresses this by using hindsight information to extract a stronger learning signal from each interaction [9], [10], [11], [12], and [13].

How does on-policy self-distillation train an SLM from hindsight? The idea is similar to how people learn from the consequences of their actions. Consider a salesperson writing outreach emails to increase customer engagement. Most recipients may never reply, but later events still provide useful clues: some may visit the product page, start a trial, or unsubscribe. None of these observations explicitly states what the salesperson should have written, yet they reveal how recipients responded and how the outreach strategy might improve. Similarly, an SLM can learn from what happens after it takes an action, without requiring a manually annotated reward or a ground-truth action.

On-policy self-distillation turns these hindsight observations into supervision by allowing the same model to reconsider its action with access to what happened afterward. This hindsight-informed prediction then becomes a training target for the model operating from the original context alone. In the following sections, we show how to implement this process on-device.

Preliminaries: Why Do Existing Fine-Tuning Methods Fall Short?

Consider an SLM that manages push notifications on an Android phone. Given a notification and the current user context, the model must decide whether to deliver it immediately, delay it, or suppress it. The goal is to surface useful information while minimizing unnecessary interruptions.

This is naturally an online learning problem. At each round \(t\), the model receives a context \(x_t\), samples an action \(y_t\) using its current parameters \(\theta_t\), observes what happens afterward as hindsight \(z_t\), and updates itself before the next interaction:

model = initialize_pretrained_model()

for t in interaction_stream:
    x_t = observe_context()
    y_t = model.sample(x_t)
    execute(y_t)

    z_t = observe_hindsight()
    model.update(x_t, y_t, z_t)

Formally, each round follows

y_t \sim \pi_{\theta_t}(\cdot \mid x_t), \qquad \theta_{t+1} = \operatorname{Update}(\theta_t; x_t,y_t,z_t).

The objective is to make increasingly better decisions over time. Let \(\ell_t(y)\) denote the latent loss of taking action \(y\) in context \(x_t\), such as the cost of interrupting the user or withholding a useful notification. We measure performance through cumulative regret relative to the best policy in a policy class \(\Pi\):

\operatorname{Regret}(T) = \sum_{t=1}^{T} \mathbb{E}_{y \sim \pi_{\theta_t}(\cdot \mid x_t)} \left[\ell_t(y)\right] - \min_{\pi \in \Pi} \sum_{t=1}^{T} \mathbb{E}_{y \sim \pi(\cdot \mid x_t)} \left[\ell_t(y)\right].

Low regret means that, over time, the adaptive model performs nearly as well as the best policy that could have been chosen in hindsight. Ideally, regret grows sublinearly with \(T\), so the average performance gap, \(\operatorname{Regret}(T)/T\), approaches zero.

The challenge is that the loss needed to minimize regret is not directly observed. After routing a notification, the device may observe whether the user opens, dismisses, or ignores it, but it does not observe the true loss \(\ell_t(y_t)\) or the outcomes of actions it did not take. The update must therefore learn from the hindsight observation \(z_t\) without treating it as a ground-truth label or a reliable scalar reward.

Supervised fine-tuning requires a ground-truth target. Let \(\pi_\theta\) denote a language model with parameters \(\theta\). Supervised fine-tuning maximizes the likelihood of a desired response:

\max_\theta \mathbb{E}_{(x,y^*) \sim \mathcal{D}} \left[ \log \pi_\theta(y^* \mid x) \right],

where \(x\) is the input context and \(y^*\) is the correct output. For notification routing, however, \(y^*\) is unavailable. We do not know whether the optimal action was to deliver a notification immediately, delay it, or suppress it entirely.

Reinforcement learning replaces the target label with a reward. Instead of learning from a ground-truth output, the model samples an action \(y\) and maximizes its expected reward:

\max_\theta \mathbb{E}_{x \sim \mathcal{D},\,y \sim \pi_\theta(\cdot \mid x)} \left[ R(x,y) \right],

where \(R(x,y)\) measures the quality of the selected action.

In practice, this reward is not directly available either. Users do not explicitly score every notification decision, and their observable behavior provides only an imperfect signal. Opening a notification does not necessarily mean that it arrived at the right time, while ignoring one does not necessarily mean that it was unimportant—the user may simply have been busy.

We therefore have neither the labels required by supervised learning nor the verifiable rewards required by reinforcement learning. At best, each interaction produces hindsight \(z_t\) that contains indirect evidence about the original decision. The central challenge is to turn this delayed and ambiguous evidence into an online update that reduces future regret.

Method: Online SDFT from Hindsight

We adapt SDFT [9] to a stream of on-device interactions. At time \(t\), the current model receives context \(x_t\) and samples an action

y_t \sim \pi_{\theta_t}(\cdot \mid x_t).

After the action is executed, the device observes hindsight information \(z_t\), such as whether the user opened, dismissed, or ignored a notification.

The hindsight-conditioned model becomes the teacher for the original decision. We run the same model with and without \(z_t\):

p_t(y)=\pi_{\theta_t}(y\mid x_t), \qquad q_t(y)=\pi_{\theta_t}(y\mid x_t,z_t).

Because \(q_t\) has access to what happened afterward, it can make a better-informed prediction about which action should have been taken.

We distill this prediction back into the model using only the original context. For each collected interaction, we minimize

\mathcal{L}_{\mathrm{SDFT}} = D_{\mathrm{KL}} \left( \operatorname{sg}\!\left[ \pi_{\theta_t}(\cdot\mid x_t,z_t) \right] \,\middle\|\, \pi_\theta(\cdot\mid x_t) \right),

where \(\operatorname{sg}\) stops gradients through the teacher. The updated model therefore learns to predict from \(x_t\) what it could previously infer only after observing \(z_t\).

The resulting learning loop is online and on-policy.

for x in interaction_stream:
    y = model.sample(x)                  # act with the current policy
    execute(y)
    z = observe_hindsight()

    teacher = stop_gradient(model(x, z))
    student = model(x)

    update(kl_divergence(student, teacher))

In a live stream, feedback may mature many decisions after the action that caused it. Each lesson was collected by a recent version of the behavior policy, but replay may reuse it after the policy has changed. No ground-truth action, explicit reward annotation, or separate teacher model is required. The benchmark below adds reliability gating and causal support to this conceptual objective so that an ambiguous callback is not mistaken for a complete label.

Two practical tricks for online SDFT

However, the original SDFT objective does not by itself specify how to collect useful hindsight from an action-dependent stream or how to train stably with a small memory budget. In online learning, the model must act before feedback exists, and its action determines which outcome can later be observed. A greedy policy can therefore reinforce its early choices and stop discovering useful alternatives. At the same time, updating on one sparse, correlated interaction at a time can make a small adapter noisy and prone to forgetting. We address these two problems with controlled exploration and bounded replay.

Trick 1: explore while uncertainty is useful, then taper. Let \(p_t(a)\) be the student’s action distribution and \(g_t\) the one-hot distribution on its most likely action. We begin with an epsilon-greedy behavior distribution

b_t^{(0)}=(1-\epsilon)g_t+\epsilon\,U(\mathcal A).

While the student’s maximum action probability remains below a configurable confidence threshold \(\tau\), we mix in a domain-specific probe action that is likely to produce an observable outcome:

\rho_t=\rho_0\,2^{-(t-1)/h_\rho}, \qquad b_t^{(1)}=(1-\rho_t)b_t^{(0)}+\rho_t\,\delta_{a_{\mathrm{probe}}}.

Otherwise, \(b_t^{(1)}=b_t^{(0)}\).

Later in the stream, the whole exploratory policy tapers toward the greedy action. With taper onset \(t_0\) and taper half-life \(h_\lambda\),

\lambda_t= \begin{cases} 1, & t\le t_0,\\ 2^{-(t-t_0)/h_\lambda}, & t>t_0, \end{cases} \qquad b_t=\lambda_t b_t^{(1)}+(1-\lambda_t)g_t, \qquad a_t\sim b_t.

The epsilon component prevents immediate lock-in, the uncertainty gate spends extra exploration only when it is informative, and the decay limits the long-run cost of probing. The probe action depends on the application: it should be an action whose eventual outcome provides useful evidence. The exploration rate, confidence threshold, taper onset, and decay schedules are configurable rather than intrinsic constants of Online-SDFT.

p = student(x)
greedy = one_hot(argmax(p))
behavior = (1 - epsilon) * greedy + epsilon * uniform(actions)

if max(p) <= confidence_threshold:
    probe = initial_probe_mass * probe_decay(t)
    behavior = (1 - probe) * behavior + probe * one_hot(feedback_action)

if t > taper_start:
    taper = taper_decay(t - taper_start)
    behavior = taper * behavior + (1 - taper) * greedy

action = sample(behavior)

Trick 2: replay a small, balanced set of recent lessons. Delayed feedback is sparse and often correlated: several similar outcomes can arrive close together. A single-example update wastes old evidence and lets common outcomes dominate. We instead retain a bounded window of the most recent usable lessons,

\mathcal B_t= \operatorname{KeepLast}_{M} \left(\mathcal B_{t-1}\cup\{(x_t,T_t,c_t)\}\right),

where \(T_t\) is the soft target and \(c_t\) is its feedback group. For an older lesson \(i\) with age \(d_i\), its sampling weight is

w_i=\eta_{c_i} \frac{2^{-d_i/h}} {\sum_{j:c_j=c_i}2^{-d_j/h}},

where \(h\) controls recency and \(\eta_c\) controls the relative weight of each feedback group. Reliable evidence receives full weight, while ambiguous evidence can be downweighted. Normalizing recency within each group prevents frequent outcomes from crowding out rare ones while still following changing behavior. Every batch includes the newest lesson, so replay does not sacrifice responsiveness. Training begins once enough usable evidence has accumulated, then uses bounded minibatches and a small fixed number of soft-target updates:

\mathcal L_{\mathcal S} =- \frac{1}{|\mathcal S|} \sum_{i\in\mathcal S}\sum_a T_i(a)\log \pi_\theta(a\mid x_i).

if feedback_produces_target:
    replay.append((x, soft_target, feedback_group))
    replay = replay[-buffer_capacity:]

if len(replay) >= minimum_replay_size:
    for _ in range(update_steps):
        batch = [replay[-1]]
        batch += weighted_sample(
            replay[:-1],
            size=remaining_batch_slots(batch_size),
            balance_by="feedback_group",
            recency_half_life=recency_half_life,
        )
        update_lora(mean_soft_cross_entropy(batch))

Replay is training-only and never enlarges the serving prompt.

Other safeguards worth mentioning:

Reliability gating and causal support skip UNKNOWN outcomes and avoid converting ambiguous evidence into a false one-hot label.
Sealed delayed-feedback bookkeeping attaches each matured outcome to the exact context, action, and decision-time distribution that produced it, preserving act-before-learn order.
A stable same-model teacher reviews hindsight with LoRA disabled, while only the small LoRA adapter is updated. This avoids a second model and prevents student updates from moving the teacher.

These safeguards make the loop causal and stable; controlled exploration and bounded replay are the main data-collection and optimization additions in the released configuration.

Experiments: Push notification routing

Notification routing is a convenient controlled test bed for delayed, action-dependent feedback: an assistant must act now, but the evidence needed to learn arrives later and only for the route it actually chose. The same structure appears in other personalization problems, including ranking suggestions, adapting shortcuts, and learning a user’s preferred writing style. The experiment should therefore be read as a test of the learning loop, not as a claim that notification routing is its only application.

Setup

The released LoRA benchmark uses the synthetic semantic-title-body-sharp-t001 dataset and LiquidAI/LFM2.5-230M. It contains three paired streams (seeds 0–2) with 240 chronological decisions per stream, for 720 decisions per method. Each stream moves through weekday, on-call, and off-hours regimes. The run used NumPy 2.4.6 on Apple MPS.

At each decision, the model sees a synthetic notification title and body, category, local time, current regime, and an on-device importance estimate. It chooses one of three actions: interrupt now, save for later, or archive. The chosen action determines which callback can later be observed. A delivered notification may be opened immediately, opened later, or deleted; a digest may be opened or deleted; an archived item produces no observable selection. Feedback that has not matured by the end of the stream is not released early.

The simulator keeps two evaluation-only quantities hidden from every method: a sampled user preference and the utility of each possible route. We report three different metrics:

Sampled-preference accuracy is the fraction of decisions matching the hidden sampled preference. Higher is better.
Cumulative regret sums the utility lost relative to the best route for each event. Lower is better.
Observable reward per decision scores only what happened on the executed delivery surface. It is the scalar seen by REINFORCE, and it is not interchangeable with either accuracy or regret.

All six methods receive the same paired event streams and are scored before learning from each event’s outcome. The three weight-updating arms—REINFORCE, RFT, and Online-SDFT—use the same reset PEFT LoRA architecture: rank 4, alpha 8, zero dropout, and 172,032 trainable parameters across 48 adapter tensors in the q_proj, k_proj, v_proj, and self_attn.out_proj modules of six attention layers. The 230-million-parameter base stays frozen, similarly named convolutional projections are not adapted, and the LoRA adapter is never merged into the base model. One physical LFM is reused sequentially, with the same initial adapter state restored before every method and seed.

Baselines

How the six methods adapt in the LoRA comparison
Method	Learning signal in the LoRA comparison
Base	Keeps the LFM frozen and never adapts.
ICL	Adds the three most recent reliable, single-route callbacks to the prompt.
RAG	Retrieves three reliable callbacks using an equal blend of metadata and visible title/body similarity.
REINFORCE	Trains the LoRA adapter from the factual scalar reward of each known callback, without replay or a hindsight teacher.
RFT (Rejection Fine-Tuning)	Draws one categorical hindsight-teacher candidate, keeps it only when reliable single-route evidence verifies it, and trains the same LoRA architecture with a hard target. This is a causal online adaptation of practical rejection fine-tuning, rather than a reproduction of the original offline RFT pipeline.
Online-SDFT	Trains the LoRA adapter from a reliability-conditioned soft target that preserves uncertainty.

For a reliable callback that supports one action, the lesson blends three signals: 5% from an adapter-disabled review by the same physical LFM, 5% from the beliefs saved when the decision was made, and 90% from the routes supported by the observed behavior. For an ambiguous digest open, the method redistributes the saved decision-time beliefs across the still-plausible actions. An unobservable (UNKNOWN) callback creates no target or update. This causal filter prevents the learner from inventing outcomes for actions that were never taken.

Results

Across the three synthetic streams, Online-SDFT had the highest mean sampled-preference accuracy and the lowest mean cumulative regret. The values below are means with nominal 95% confidence intervals across only three paired streams.

LoRA results and nominal 95% confidence intervals across three paired synthetic streams (n = 3)
Method	Sampled-preference accuracy	Cumulative regret ↓	Observable reward / decision
Base	28.19% ± 2.33	164.67 ± 2.66	0.256 ± 0.140
ICL	42.22% ± 3.57	131.48 ± 0.60	0.008 ± 0.055
RAG	50.00% ± 7.37	106.62 ± 30.45	0.316 ± 0.389
REINFORCE	32.08% ± 2.06	157.99 ± 2.56	-0.090 ± 0.176
RFT	52.78% ± 2.60	105.26 ± 15.32	0.364 ± 0.250
Online-SDFT	70.28% ± 2.60	44.76 ± 4.57	0.955 ± 0.082

Online-SDFT made 506 of 720 decisions that matched the sampled preference. Compared with RFT, the strongest baseline by both mean accuracy and mean cumulative regret, it improved accuracy by 17.50 percentage points and reduced cumulative regret by 60.50. RFT accepted only 75 of 311 teacher candidates (24.12%), illustrating the data-efficiency cost of converting hindsight into a verified hard label; the soft target can retain graded information from more usable callbacks.

Ablation studies

To isolate the two online-learning tricks, we reran Online-SDFT as a paired \(2\times2\) ablation. Every variant keeps the same hindsight target, LoRA architecture, optimizer, two update steps, four-lesson warmup, and event streams. The no-replay variants use only the newest lesson in both optimizer steps.

Online-SDFT exploration and replay ablation
Variant	Serving policy	Update data	Sampled-preference accuracy ↑	Cumulative regret ↓
Full Online-SDFT	Controlled exploration	64-row balanced replay	70.28% ± 2.60	44.76 ± 4.57
No exploration	Greedy	Same replay	69.44% ± 4.72	62.00 ± 10.96
No replay	Same exploration	Newest lesson only	39.58% ± 1.70	134.90 ± 11.34
Neither trick	Greedy	Newest lesson only	42.78% ± 3.41	130.42 ± 4.12

Replay made the largest difference on these streams. Removing it reduced mean accuracy by 30.69 percentage points and added 90.14 cumulative regret. Removing exploration while retaining replay changed accuracy by only 0.83 points, but added 17.24 regret: greedy serving still matched many sampled preferences, yet made more costly mistakes. Values are means with nominal 95% confidence intervals across the same three paired synthetic streams.

Online-SDFT on Android

The repository includes an Android proof of concept that brings the learning loop onto one device. A local LiquidAI/LFM2.5-230M makes each decision; once hindsight becomes available, ONNX Runtime Training updates a rank-4 LoRA adapter while the base model remains frozen. Replay and adapter checkpoints stay in app-private storage and survive restarts.

On a physical phone, no network. The same notification is dismissed from Android’s shade until the model keeps it quiet and files it under Saved. The person then asks for it back, and the next one is delivered again: the correction moved Show now from silenced to 94%. Every signal comes from the shade gestures, and the adapter update runs on the device.

Model export and provisioning still happen on a Linux host, and the current FP32 graph targets high-memory ARM64 devices. The complete Android deployment guide contains the exact requirements, build and provisioning commands, Android permissions, verification logs, reset procedure, and failure checks.

Engineering prototype. This demonstrates the Android integration for real on-device LoRA updates; it does not yet establish production latency, battery use, or thermal behavior.

Conclusion and limitations

This controlled experiment supports a focused hypothesis: when feedback is delayed, partial, and generated by the model’s own actions, a reliability-conditioned soft hindsight target can be more useful than prompt memory, a scalar reward, or a rejection-filtered hard target. The broader idea is not specific to notifications. A compact model can act on current context, revisit a completed interaction with additional factual evidence, and distill that lesson into a small local update for future decisions.

The evidence is still narrow. The benchmark uses three synthetic streams rather than real users, and the selected Online-SDFT configuration has not been confirmed on held-out streams. The run used Apple MPS and did not measure Android latency, peak memory, battery use, thermal throttling, background-job reliability, or end-to-end privacy. Small language models also remain constrained by reasoning ability and context length. The next step is therefore a larger held-out comparison followed by profiling on physical edge devices.

References

Marah Abdin et al. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv:2404.14219, 2024.
Meta AI. Introducing Quantized Llama Models with Increased Speed and a Reduced Memory Footprint. 2024.
Google AI for Developers. Gemma 3n Model Overview. 2025.
Alexander Amini et al. LFM2 Technical Report. arXiv:2511.23404, 2025.
Huxiao Ji et al. TIM: Temporal Interaction Model in Notification System. Proceedings of the International Conference on Multimedia Retrieval, 2024.
Yiping Yuan, Ajith Muralidharan, Preetam Nandy, Miao Cheng, and Prakruthi Prabhakar. Offline Reinforcement Learning for Mobile Notifications. Proceedings of the ACM International Conference on Information and Knowledge Management, 2022.
Charles Packer et al. MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560, 2023.
Ruiyang Qin et al. Enabling On-Device Large Language Model Personalization with Self-Supervised Data Selection and Synthesis. Proceedings of the ACM/IEEE Design Automation Conference, 2024.
Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-Distillation Enables Continual Learning. arXiv:2601.19897, 2026.
Siyan Zhao et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv:2601.18734, 2026.
Jonas Hübotter et al. Reinforcement Learning via Self-Distillation. arXiv:2601.20802, 2026.
Thomas Kleine Buening, Jonas Hübotter, Barna Pásztor, Idan Shenfeld, Giorgia Ramponi, and Andreas Krause. Aligning Language Models from User Interactions. arXiv:2603.12273, 2026.
Woongyeng Yeo, Yumin Choi, Taekyung Ki, and Sung Ju Hwang. HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents. arXiv:2605.17873, 2026.