Shared Pretrained Encoder
CLIP-pretrained ViT-L/14 reference encoder.
Reinforcement Learning Final Project 2026
MMCR
Sequential Layer-wise Model Merging via Reinforcement Learning
National Yang Ming Chiao Tung University
We turn model merging into a sequential decision problem: an RL policy visits each layer, observes task-vector geometry and coefficient history, then chooses how much each source model should contribute to the merged model.
What Are We Doing?
Model merging combines several task-specific models into one unified model without retraining the full network. It is useful for multi-task learning and efficient deployment, but it becomes difficult when task vectors point in conflicting directions.
MMCR studies whether reinforcement learning can learn better layer-wise merging decisions than fixed rules. Instead of applying one global coefficient to the whole model, the policy chooses an adaptive coefficient vector for each mergeable layer.
86.41%
MMCR average accuracy
+4.78pp
points over TIES baseline
~1.57x
final two-GPU wall-clock speedup
Why This Is Hard
Layer-wise Conflict
Different tasks may update the same layer with different magnitudes or directions, so a fixed merging rule can be too coarse.
Sequential Dependency
Early layer decisions affect the partially merged model and can change what later layer decisions should look like.
Credit Assignment
The merged model is usually evaluated after multiple layers, making it hard to know which coefficient choices helped most.
MMCR Pipeline
Task-specific Encoders
Downloaded fine-tuned encoders with trained task heads.
Task-vector Construction
Compute encoder task vectors from model differences.
TIES Preprocessing
Remove weak updates and reduce sign conflicts before RL search.
↓
5
Environment Setup
Precompute Geometry
Norms, cosine similarity, layer statistics.
Construct Reward Subsets
64-sample entropy batches per task.
Initialize Layer-wise MDP
States, actions, rewards, coefficient history.
↓
6
Sequential Layer-wise Merging Episode
State st
Layer progress, current-layer task-vector geometry, the full coefficient table, and an updated-layer mask.
Action at
A raw Gaussian action is transformed into non-negative coefficients for the K source task vectors.
Update Layer
Merge only the current layer.
Reward rt
Negative prediction entropy on small reward subsets, plus a standard-deviation penalty for task balance.
Next State
Advance until the terminal layer.
Visit layers sequentially, collect returns, then update the policy.
↓
GRPO + RLOO
GRPO updates the policy from groups of full merging trajectories. RLOO uses the other trajectories as a leave-one-out baseline, so advantages are relative and no critic is needed.
Selection
The best coefficient table observed during training constructs the final merged encoder.
Final Evaluation
Evaluate task heads on full test splits.
Project Details
Problem formulation
Given K fine-tuned source models and one shared pretrained model, MMCR represents each source model as a task vector. For layer l and source model k, the task vector is the parameter difference between the fine-tuned model and the pretrained backbone. The merged layer is constructed by adding a weighted sum of task vectors back to the pretrained layer.
Task vector of source model k at layer l.
Pretrained backbone parameter at layer l.
Fine-tuned source model k parameter at layer l.
Learned merging coefficient for source model k at layer l.
Sequential layer-wise MDP
One episode corresponds to constructing one complete merged model. At each step, the environment focuses on one mergeable layer, writes the policy-selected coefficient vector into that layer, updates the coefficient table, and advances to the next layer. This turns model merging from a one-shot coefficient search into a trajectory-level optimization problem.
State dimension
1 + 2K + LK + L, using progress, geometry, coefficient history, and filled mask.
Action space
K-dimensional coefficient vector for the current layer.
Transition
Deterministic layer update; all other layers remain unchanged at that step.
Objective
Maximize cumulative reward over the full layer-wise merging trajectory.
Policy optimization
MMCR uses a stochastic Gaussian policy and optimizes it with a GRPO-style objective. Each update samples a group of complete merging trajectories. Instead of learning a separate value function, the method computes a return leave-one-out baseline from the other trajectories in the same group.
Number of trajectories sampled in one policy update group.
Return of trajectory i after constructing and evaluating a merged model.
Leave-one-out baseline computed from the other trajectories.
Normalized advantage used to update the policy.
Experimental Setup
Benchmark Tasks
SUN397, Stanford Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, and DTD.
Model Setting
Vision Transformer model merging with task-specific fine-tuned source models and a shared pretrained
backbone.
Optimization Budget
MMCR uses 300 iterations, group size 8, positive coefficient transformation, and intermediate entropy
reward.
Reward Budget
Policy optimization uses reward subsets of 64 samples per task; final accuracy is measured on the full
evaluation protocol.
Hardware
Experiments run on a workstation with two NVIDIA RTX 3090 GPUs.
The compared model merging baselines include Task Arithmetic, TIES, DARE-TA, NAN-TA, and AdaMerging. For methods that require data during coefficient optimization, the reward or validation sample budget is aligned with MMCR whenever possible, so a method does not gain advantage simply by using more optimization samples.
Main Results
MMCR achieves the best average accuracy among the compared merging methods and remains competitive against longer-running AdaMerging baselines.
| Method | SUN397 | Cars | RESISC45 | EuroSAT | SVHN | GTSRB | MNIST | DTD | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| TIES | 76.11 | 83.93 | 83.65 | 80.66 | 84.70 | 89.14 | 89.29 | 65.53 | 81.63 |
| AdaMerging-300 epochs | 78.20 | 85.56 | 87.67 | 83.50 | 68.45 | 86.81 | 61.74 | 70.96 | 77.86 |
| AdaMerging-500 epochs | 78.71 | 87.58 | 90.21 | 88.14 | 88.13 | 89.37 | 91.45 | 72.87 | 85.81 |
| AdaMerging-600 epochs | 78.70 | 87.66 | 90.62 | 88.78 | 89.08 | 89.79 | 92.61 | 73.67 | 86.36 |
| MMCR (ours) | 77.57 | 89.11 | 89.06 | 85.72 | 90.65 | 89.96 | 92.54 | 76.64 | 86.41 |
Against TIES
MMCR improves average accuracy from 81.63% to 86.41%, showing that learned layer-wise coefficients add value beyond preprocessing alone.
Against AdaMerging
MMCR slightly outperforms AdaMerging-600 epochs while using a trajectory-based optimization structure that parallelizes naturally.
Comparison with model merging Baseline
| Method | Avg. Accuracy | Role in the experiment |
|---|---|---|
| Pretrained model | 65.12 | Lower reference before task-specific fine-tuning. |
| Fine-tuned source models | 93.06 | Upper reference before compression into one merged model. |
| Task Arithmetic (0.3) | 78.67 | Simple linear task-vector merging baseline. |
| DARE-TA | 78.29 | Task arithmetic with dropped and rescaled updates. |
| NAN-TA | 73.37 | Training-free coefficient estimation baseline. |
| TIES | 81.63 | Strong data-free task-vector conflict reduction baseline. |
| AdaMerging-500 epochs | 85.81 | Strong data-based coefficient optimization baseline. |
| AdaMerging-600 epochs | 86.36 | Longer data-based coefficient optimization baseline. |
| MMCR (ours) | 86.41 | Sequential RL-based layer-wise merging with TIES preprocessing and GRPO + RLOO. |
Runtime and Parallelization
| Method | GPUs Used | Runtime | Time (min) | Avg. Acc. |
|---|---|---|---|---|
| TIES | No (CPU) | 3m 14s | 3.23 | 81.63 |
| AdaMerging-300 epochs | 1 GPU | 3h 05m 00s | 185.00 | 77.86 |
| AdaMerging-500 epochs | 1 GPU | 5h 56m 52s | 356.87 | 85.81 |
| AdaMerging-600 epochs | 1 GPU | 6h 58m 54s | 418.90 | 86.36 |
| MMCR (ours) | 1 GPU | 4h 05m 10s | 245.17 | 86.41 |
| MMCR (ours) | 2 GPUs | 2h 35m 58s | 155.97 | 86.41 |
The final two-GPU MMCR run gives about a 1.57x wall-clock speedup. The gain is useful, while synchronization and policy-update overhead prevent perfect scaling.
Ablation Studies
TIES Preprocessing
Using TIES improves MMCR from 83.69% to 86.41% average accuracy. The largest gains appear on MNIST and EuroSAT.
RL Optimizer
GRPO + RLOO reaches 86.41%, while PPO reaches 78.19%. Group-relative advantages are more stable for this sparse, expensive reward setting.
Reward Frequency
Terminal-only reward is efficient at 86.23%, while interval 6 reaches 86.57% with higher runtime. Reward density improves accuracy but costs more.
Policy Transferability
Transfer works well for related two-task groups, but becomes weaker for three-task or unrelated transfers, suggesting task similarity matters.
TIES preprocessing
| Setting | Avg. Accuracy | Conclusion |
|---|---|---|
| MMCR without TIES | 83.69 | Raw task vectors contain noisy or conflicting updates. |
| MMCR with TIES | 86.41 | TIES improves the search space and gives a +2.72 point gain. |
Policy optimization algorithms
| Method | Avg. Accuracy | Time (min) | Interpretation |
|---|---|---|---|
| PPO | 78.19 | 24.50 | Fast, but unstable in this sparse reward setting. |
| Dr. GRPO | 82.73 | 148.47 | Removing group standard-deviation normalization hurts performance. |
| DAPO-lite | 86.05 | 393.93 | Competitive but too expensive due to resampling overhead. |
| GSPO | 86.32 | 151.18 | Very close to GRPO, using trajectory-level policy ratios. |
| MMCR | 86.41 | 155.97 | Best accuracy and selected as the final MMCR optimizer. |
Reward evaluation frequency
| Reward Setting | Avg. Accuracy | Time (min) | Conclusion |
|---|---|---|---|
| Terminal-only | 86.23 | 50.40 | Most efficient; final reward is already informative. |
| Interval 13 | 86.41 | 155.97 | Final setting; balances reward density and runtime. |
| Interval 6 | 86.57 | 247.37 | Slightly higher accuracy but more expensive. |
Policy transferability
Related 2-task
EuroSAT/SVHN → RESISC45/MNIST
- TIES
- 95.20
- Transferred
- 97.33
- Target
- 97.54
Related 3-task
EuroSAT/SVHN/DTD → RESISC45/MNIST/GTSRB
- TIES
- 93.20
- Transferred
- 91.05
- Target
- 96.26
Unrelated 2-task
EuroSAT/RESISC45 → MNIST/SVHN
- TIES
- 96.97
- Transferred
- 94.02
- Target
- 97.06
Conclusions
What worked
Formulating model merging as a sequential layer-wise RL problem lets the policy adapt coefficients across layers instead of relying on a single fixed merging rule.
Best final method
The strongest configuration combines TIES-preprocessed task vectors, positive coefficient transformation, entropy-based reward, and GRPO with return leave-one-out advantages.
Main result
MMCR reaches 86.41% average accuracy across eight vision tasks, improving over TIES and slightly outperforming the longer-running AdaMerging-600 epochs baseline.
Main limitation
Reward evaluation is still expensive because each trajectory requires repeatedly constructing and evaluating merged models. Multi-GPU parallelism helps, and the final speedup is about 1.57x rather than ideal scaling. Policy transfer also becomes weaker for three-task or unrelated target groups.
Future work
Future experiments could explore cheaper reward proxies, stronger transfer adaptation, larger task groups, and improved scheduling for intermediate reward evaluation. The transfer results suggest that reusable merging policies are promising for related tasks, but target-specific optimization is still important for unrelated domains or larger task groups.