Reassessing Layer Pruning in LLMs: New Insights and Methods (2024)

Yao Lu¹ Hao Cheng Yujie Fang¹ Zeyu Wang¹ Jiaheng Wei²
Dongwei Xu¹ Qi Xuan¹ Xiaoniu Yang¹ Zhaowei Zhu
¹Zhejiang University of Technology ²HKUST-GZyaolu.zjut@gmail.com. Equal contribution with Hao Cheng.Corresponding author: xuanqi@zjut.edu.cn.

Abstract

Although large language models (LLMs) have achieved remarkable success across various domains, their considerable scale necessitates substantial computational resources, posing significant challenges for deployment in resource-constrained environments. Layer pruning, as a simple yet effective compression method, removes layers of a model directly, reducing computational overhead. However, what are the best practices for layer pruning in LLMs? Are sophisticated layer selection metrics truly effective? Does the LoRA (Low-Rank Approximation) family, widely regarded as a leading method for pruned model fine-tuning, truly meet expectations when applied to post-pruning fine-tuning? To answer these questions, we dedicate thousands of GPU hours to benchmarking layer pruning in LLMs and gaining insights across multiple dimensions. Our results demonstrate that a simple approach, i.e., pruning the final 25% of layers followed by fine-tuning the lm_head and the remaining last three layer, yields remarkably strong performance. Following this guide, we prune Llama-3.1-8B-It and obtain a model that outperforms many popular LLMs of similar size, such as ChatGLM2-6B, Vicuna-7B-v1.5, Qwen1.5-7B and Baichuan2-7B.We release the optimal model weights on Huggingface¹¹1https://huggingface.co/YaoLuzjut/Llama-3.1-6.3B-It-Alpaca and https://huggingface.co/YaoLuzjut/Llama-3.1-6.3B-It-Dolly, and the code is available on GitHub²²2https://github.com/yaolu-zjut/Navigation-LLM-layer-pruning.

1 Introduction

In recent years, large language models (LLMs) have achieved unprecedented success in many fields, such as text generation(Achiam etal., 2023; Touvron etal., 2023), semantic analysis(Deng etal., 2023; Zhang etal., 2023b) and machine translation(Zhang etal., 2023a; Wang etal., 2023). However, these achievements come with massive resource consumption, posing significant challenges for deployment on resource-constrained devices. To address these challenges, numerous techniques have been developed to create more efficient LLMs, including pruning(Ma etal., 2023a; Sun etal., 2023), knowledge distillation(Xu etal., 2024; Gu etal., 2024), quantization(Lin etal., 2024; Liu etal., 2023), low-rank factorization(Saha etal., 2023; Zhao etal., 2024a), and system-level inference acceleration(Shah etal., 2024; Lee etal., 2024).

Among these methods, pruning has emerged as a promising solution to mitigate the resource demands of LLMs. By selectively removing redundant patterns—such as parameters(Sun etal., 2023), attention heads(Ma etal., 2023a) and layers(Men etal., 2024)—pruning aims to slim down the model while maintaining its original performance as much as possible. Among different types of pruning, layer pruning(Kim etal., 2024; Siddiqui etal., 2024) has garnered particular interest due to its direct impact on pruning the model’s depth, thereby decreasing both computational complexity and memory usage. Additionally, thanks to the nice structure of the existing LLMs such as Llama(Dubey etal., 2024), whose transformer blocks have the exactly same dimension of input and output, layer pruning becomes a straightforward and simple solution. Therefore, in this paper, we focus on layer pruning. Unlike existing studies(Men etal., 2024; Yang etal., 2024b; Chen etal., 2024; Zhong etal., 2024; Liu etal., 2024b) that aim to propose various sophisticated pruning methods, we take a step back and focus on the following questions:

Reassessing Layer Pruning in LLMs: New Insights and Methods (1)

Reassessing Layer Pruning in LLMs: New Insights and Methods (2)

Q1.
Layer Selection: Are fancy metrics essential for identifying redundant layers to prune?
Q2.
Fine-Tuning: Is the LoRA family the best choice for post-pruning fine-tuning?
Q3.
Pruning Strategy: Will iterative pruning outperform one-shot pruning?

To answer the aforementioned questions, we spent thousands of GPU hours to benchmark layer pruning, conducting extensive experiments across 7 layer selection metrics, 4 state-of-the-art open-source LLMs, 6 fine-tuning methods, 5 pruning strategies on 10 common datasets. From these efforts, we have developed a practical list of key insights for LLM layer pruning in Figure1:

1).
Reverse-order pruning is simple yet effective, i.e., simply pruning the last several layers performs better than many complex pruning metrics(Kim etal., 2024; Men etal., 2024) .
2).
LoRA performs worse than expected, i.e., LoRA, the most commonly used fine-tuning methods in existing pruning approaches(Sun etal., 2023; Ma etal., 2023b; Kim etal., 2024; Men etal., 2024), is not the best choice for post-pruning performance recovery. In contrast, freezing the other layers and fine-tuning only the last few remaining layers and lm_head, also known as partial-layer fine-tuning, can achieve higher accuracy while reducing the training time. The result is unique to layer pruning since LoRA and partial-layer fine-tuning perform similarly as Table3 in full-model fine-tuning.
3).
Iterative pruning offers no benefit, i.e., considering both training costs and performance gains, iterative pruning, where layers are removed step-by-step, fails to beat the one-shot pruning, where a single cut is made.

In addition to the above practices, we also conduct sensitivity analyses on the number of calibration samples, the choice of Supervised Fine-Tuning (SFT) datasets and various pruning rates for LLM layer pruning. We find that the number of calibration samples affects the performance of data-driven pruning methods, highlighting the importance of considering performance stability as a key criterion when evaluating the quality of pruning metrics. Similarly, we discover that fine-tuning with different SFT datasets significantly impacts the performance of pruned models. This suggests the need for further exploration of the most suitable datasets for fine-tuning. Finally, we apply our insights and practices to prune Llama-3.1-8B-Instruct(Dubey etal., 2024), obtaining Llama-3.1-6.3B-It-Alpaca and Llama-3.1-6.3B-It-Dolly, as shown in Figure1. These pruned models require significantly fewer training tokens but outperform several popular community LLMs of similar size, such as ChatGLM2-6B(GLM etal., 2024), Vicuna-7B-v1.5(Zheng etal., 2024), Qwen1.5-7B(Yang etal., 2024a) and Baichuan2-7B(Baichuan, 2023). We hope our work will help guide future efforts in LLM layer pruning and inform best practices for deploying LLMs in real-world applications. In a nutshell, we make the following contributions:

•
Comprehensive Benchmarking: We conduct an extensive evaluation of layer selection metrics, fine-tuning methods, and pruning strategies, providing practical insights into effective pruning techniques based on thousands of GPU hours across multiple datasets.
•
Novel Best Practices: We identify reverse-order as a simple and effective layer selection metric, find that partial-layer fine-tuning outperforms LoRA-based techniques, and demonstrate that one-shot pruning is as effective as iterative pruning while reducing training costs.
•
Optimized Pruned LLMs: We release Llama-3.1-6.3B-It-Alpaca and Llama-3.1-6.3B-It-Dolly, which are obtained through direct pruning of the Llama-3.1-8B-Instruct. Our pruned models require up to $10^{6}\times$ fewer training tokens compared to training from scratch, while still comparing favorably to various popular community LLMs of similar size, such as ChatGLM2-6B(GLM etal., 2024), Vicuna-7B-v1.5(Zheng etal., 2024), Qwen1.5-7B(Yang etal., 2024a) and Baichuan2-7B(Baichuan, 2023).

2 Related Work

LLM Layer Pruning. LLM layer pruning is a technique used to reduce the number of layers in LLMs, aiming to lower computational costs without significantly degrading performance. Specifically, it evaluates the contribution of each layer to the model’s overall performance, using criteria such as gradients, activation values, parameter weights, or the layer’s influence on the loss function. Layers that contribute the least are then pruned to reduce complexity. For example, LaCo(Yang etal., 2024b) achieves rapid model size reduction by folding subsequent layers into the previous layer, effectively preserving the model structure. Similarly, MKA(Liu etal., 2024b) uses manifold learning and the Normalized Pairwise Information Bottleneck measure(Tishby etal., 2000) to identify the most similar layers for merging. ShortGPT(Men etal., 2024) uses Block Influence (BI) to measure the importance of each layer in LLMs and remove layers with low BI scores. Kim etal. (2024) utilize Magnitude, Taylor and Perplexity (PPL) to evaluate the significance of each layer.

3 Background and Notation

3.1 Problem Formulation for Layer Pruning

An LLM $\mathcal{M}$ consists of multiple Transformer layers $L=\{l_{1},l_{2},\cdots,l_{n}\}$ , each containing a pair of multi-head attention and feed-forward network modules:

\displaystyle\mathcal{M}

\displaystyle=l_{1}\circ l_{2}\cdots\circ l_{n},

(1)

Layer pruning aims to find a subset of layers $L^{\prime}\subseteq L$ such that the pruned model $\mathcal{M}^{\prime}$ maintains acceptable performance while reducing the model’s complexity, which can be formalized as:

	Minimize	$\displaystyle\mathcal{C}\left(\mathcal{M}^{\prime}\right),$		(2)
	$\displaystyle\text{s.t.}\quad P\left(\mathcal{M}^{\prime}\right)\geq\alpha$	$\displaystyle\times P(\mathcal{M}),L^{\prime}\subseteq L,$		(2)

where $\mathcal{C}\left(\mathcal{M}^{\prime}\right)$ denotes the complexity of the pruned model, which can be quantified in terms of the number of parameters, FLOPs, or inference time, etc. $\alpha$ is a hyperparameter (e.g., $\alpha=0.9$ ) that defines the acceptable performance degradation. $P(\cdot)$ represents the performance on given tasks. Numerous methods have proposed various metrics to identify and prune unimportant layers. Herein, we include 7 popular metrics:

Random Selection. For the random selection baseline, we randomly select several layers to prune.

Reverse-order. This metric (Men etal., 2024) posits that importance is inversely proportional to the sequence order. It assigns lower importance scores to the deeper layers and prune them.

Magnitude. It was first introduced by Li etal. (2016) and subsequently adopted by Kim etal. (2024), which assumes that weights exhibiting smaller magnitudes are deemed less informative. Following Kim etal. (2024), we compute $I_{\text{Magnitude}}^{n}=\sum_{k}||W_{k}^{n}||_{p}$ , where $W_{k}^{n}$ denotes the weight matrix of operation $k$ within the $n$ -th transformer layer. In this paper, we uniformly set $p=\{1,2\}$ . As a result, we term these methods as Magnitude-l1 and Magnitude-l2.

Taylor. For a given calibration dataset $D$ , the significance of removing weight parameters is indicated by the change in training loss $\mathcal{L}:=|\mathcal{L}(W_{k}^{n},D)-\mathcal{L}(W_{k}^{n}=0,D)|\approx|%\frac{\partial\mathcal{L}(D)}{\partial W_{k}^{n}}W_{k}^{n}|$ . Following Ma etal. (2023a); Kim etal. (2024), we omit the second-order derivatives in this assessment. Then we define the Taylor score of the $n$ -th transformer layer as $I_{\text{Taylor}}^{n}=\sum_{k}{|\frac{\partial\mathcal{L}(D)}{\partial W_{k}^{%n}}W_{k}^{n}|}$ .

PPL. Following Kim etal. (2024), we remove a single layer and assess its impact on the perplexity of the pruned model using the calibration dataset $D$ . We then prune those layers that lead to a smaller degradation of the PPL.

BI. Men etal. (2024) introduce a metric called Block Influence as an effective indicator of layer importance. Specifically, the BI score of the $i$ -th layer can be calculated as follows:

\mathrm{BI}_{i}=1-\mathbb{E}_{X,t}\frac{X_{i,t}^{T}X_{i+1,t}}{\left\|X_{i,t}%\right\|_{2}\left\|X_{i+1,t}\right\|_{2}},

(3)

where $X_{i}$ denotes the input of the $i$ -th layer and $X_{i,t}$ is the $t$ -th row of $X_{i}$ .

3.2 Evaluation and Datasets

To assess the performance of the model, we follow the evaluation of Ma etal. (2023a) to perform zero-shot task classification on $8$ common sense reasoning datasets using the lm-evaluation-harness(Gao etal., 2023) package: MMLU(Hendrycks etal., 2021), CMMLU(Li etal., 2023), PIQA(Bisk etal., 2020), HellaSwag(Zellers etal., 2019), WinoGrande(Sakaguchi etal., 2021), ARC-easy(Clark etal., 2018), ARC-challenge(Clark etal., 2018) and OpenbookQA(Mihaylov etal., 2018). Additionally, we evaluate the model using perplexity on the WikiText2(Merity etal., 2016) and Penn Treebank (PTB)(Marcus etal., 1993) datasets. For the PPL metric, we follow (Ma etal., 2023a; Muralidharan etal., 2024) and use WikiText2 for calculation. Following (Ma etal., 2023a), we randomly select 10 samples from BookCorpus(Zhu etal., 2015) to compute Taylor and BI, truncating each sample to a sequence length of 128. Unless otherwise specified, we utilize the Alpaca-cleaned(Taori etal., 2023) with LoRA to recover the performance. Uniformly, we set the training epoch to 2 and batch size to 64. All experiments are conducted on 2 NVIDIA A100 GPUs with 40 GB of memory and 4 NVIDIA RTX A5000 GPUs with 24 GB of memory.

4 An Empirical Exploration of LLM Layer Pruning

This paper aims to contribute to the community the best practice of layer pruning such that practitioners can prune an LLM to an affordable size and desired performance with minimal exploration effort. Specifically, we will expand from three aspects: First, we explore which metric is most effective for identifying unimportant layers, helping researchers make informed choices.Then, we investigate which fine-tuning method most effectively restores model performance after pruning. Finally, we delve deeper into various pruning strategies and want to answer whether iterative pruning will outperform one-shot pruning.

Model	Metric	Benchmarks								Avg Acc
Model	Metric	PIQA	HellaSwag	OpenbookQA	ARC-e	ARC-c	MMLU	CMMLU	WinoGrande	Avg Acc
Vicuna-7B-v1.5	Dense	0.7720 $\pm$ 0.0098	0.5642 $\pm$ 0.0049	0.3300 $\pm$ 0.0210	0.7555 $\pm$ 0.0088	0.4326 $\pm$ 0.0145	0.4858 $\pm$ 0.0040	0.3518 $\pm$ 0.0044	0.6953 $\pm$ 0.0129	0.5484
	Reverse-order	0.7171 $\pm$ 0.0105	0.5005 $\pm$ 0.0050	0.2608 $\pm$ 0.0198	0.6221 $\pm$ 0.0099	0.3848 $\pm$ 0.0142	0.4737 $\pm$ 0.0041	0.3417 $\pm$ 0.0044	0.6267 $\pm$ 0.0136	0.4909
	Random	0.5223 $\pm$ 0.0117	0.2607 $\pm$ 0.0044	0.1380 $\pm$ 0.0154	0.2614 $\pm$ 0.0090	0.2176 $\pm$ 0.0121	0.2295 $\pm$ 0.0035	0.2500 $\pm$ 0.0040	0.4672 $\pm$ 0.0140	0.2933
	PPL	0.7361 $\pm$ 0.0103	0.4734 $\pm$ 0.0050	0.2760 $\pm$ 0.0200	0.6705 $\pm$ 0.0096	0.3456 $\pm$ 0.0139	0.2943 $\pm$ 0.0038	0.2569 $\pm$ 0.0041	0.5896 $\pm$ 0.0138	0.4553
	Magnitude-l1	0.5299 $\pm$ 0.0116	0.2586 $\pm$ 0.0044	0.1440 $\pm$ 0.0157	0.2609 $\pm$ 0.0090	0.2253 $\pm$ 0.0122	0.2297 $\pm$ 0.0035	0.2514 $\pm$ 0.0040	0.4893 $\pm$ 0.0140	0.2986
	Magnitude-l2	0.5256 $\pm$ 0.0117	0.2578 $\pm$ 0.0044	0.1340 $\pm$ 0.0152	0.2622 $\pm$ 0.0090	0.2108 $\pm$ 0.0119	0.2295 $\pm$ 0.0035	0.2515 $\pm$ 0.0040	0.4838 $\pm$ 0.0140	0.2944
	BI	0.6910 $\pm$ 0.0108	0.3987 $\pm$ 0.0049	0.2100 $\pm$ 0.0182	0.5829 $\pm$ 0.0101	0.2654 $\pm$ 0.0129	0.2389 $\pm$ 0.0036	0.2513 $\pm$ 0.0040	0.5036 $\pm$ 0.0141	0.3927
	Taylor	0.5250 $\pm$ 0.0117	0.2581 $\pm$ 0.0044	0.1360 $\pm$ 0.0153	0.2584 $\pm$ 0.0090	0.2048 $\pm$ 0.0118	0.2318 $\pm$ 0.0036	0.2526 $\pm$ 0.0040	0.4972 $\pm$ 0.0141	0.2955
Qwen1.5-7B	Dense	0.7845 $\pm$ 0.0096	0.5785 $\pm$ 0.0049	0.3160 $\pm$ 0.0208	0.7125 $\pm$ 0.0093	0.4053 $\pm$ 0.0143	0.5967 $\pm$ 0.0039	0.7277 $\pm$ 0.0039	0.6575 $\pm$ 0.0133	0.5973
	Reverse-order	0.6942 $\pm$ 0.0107	0.4444 $\pm$ 0.0050	0.2280 $\pm$ 0.0188	0.5143 $\pm$ 0.0103	0.3302 $\pm$ 0.0137	0.5101 $\pm$ 0.0041	0.7171 $\pm$ 0.0040	0.5912 $\pm$ 0.0138	0.5037
	Random	0.5408 $\pm$ 0.0116	0.2682 $\pm$ 0.0044	0.1240 $\pm$ 0.0148	0.2630 $\pm$ 0.0090	0.2039 $\pm$ 0.0118	0.2366 $\pm$ 0.0076	0.2457 $\pm$ 0.0040	0.4807 $\pm$ 0.0140	0.2954
	PPL	0.7089 $\pm$ 0.0106	0.4195 $\pm$ 0.0049	0.2240 $\pm$ 0.0187	0.5960 $\pm$ 0.0101	0.2944 $\pm$ 0.0133	0.2457 $\pm$ 0.0036	0.2552 $\pm$ 0.0041	0.5185 $\pm$ 0.0140	0.4078
	Magnitude-l1	0.6578 $\pm$ 0.0111	0.3989 $\pm$ 0.0049	0.2040 $\pm$ 0.0180	0.5244 $\pm$ 0.0102	0.2901 $\pm$ 0.0133	0.2574 $\pm$ 0.0037	0.2541 $\pm$ 0.0041	0.5249 $\pm$ 0.0140	0.3890
	Magnitude-l2	0.5903 $\pm$ 0.0115	0.3657 $\pm$ 0.0048	0.1640 $\pm$ 0.0166	0.4630 $\pm$ 0.0102	0.2381 $\pm$ 0.0124	0.2502 $\pm$ 0.0037	0.2513 $\pm$ 0.0040	0.5312 $\pm$ 0.0140	0.3567
	BI	0.7220 $\pm$ 0.0105	0.4190 $\pm$ 0.0049	0.2440 $\pm$ 0.0192	0.5972 $\pm$ 0.0101	0.2671 $\pm$ 0.0129	0.2456 $\pm$ 0.0036	0.2536 $\pm$ 0.0040	0.5383 $\pm$ 0.0140	0.4190
	Taylor	0.6970 $\pm$ 0.0107	0.4284 $\pm$ 0.0049	0.2060 $\pm$ 0.0181	0.5160 $\pm$ 0.0103	0.3140 $\pm$ 0.0136	0.5231 $\pm$ 0.0041	0.6079 $\pm$ 0.0043	0.6046 $\pm$ 0.0137	0.4871
Gemma2-2B-It	Dense	0.7867 $\pm$ 0.0096	0.5367 $\pm$ 0.0050	0.3560 $\pm$ 0.0214	0.8085 $\pm$ 0.0081	0.5111 $\pm$ 0.0146	0.5687 $\pm$ 0.0039	0.4499 $\pm$ 0.0045	0.6961 $\pm$ 0.0129	0.5892
	Reverse-order	0.7029 $\pm$ 0.0107	0.4529 $\pm$ 0.0050	0.2660 $\pm$ 0.0198	0.6343 $\pm$ 0.0099	0.3763 $\pm$ 0.0142	0.5261 $\pm$ 0.0040	0.4117 $\pm$ 0.0045	0.6551 $\pm$ 0.0134	0.5032
	Random	0.7307 $\pm$ 0.0104	0.4462 $\pm$ 0.0050	0.2860 $\pm$ 0.0202	0.6852 $\pm$ 0.0095	0.3422 $\pm$ 0.0139	0.3452 $\pm$ 0.0040	0.2893 $\pm$ 0.0042	0.5833 $\pm$ 0.0139	0.4635
	PPL	0.7454 $\pm$ 0.0102	0.4611 $\pm$ 0.0050	0.2940 $\pm$ 0.0204	0.7008 $\pm$ 0.0094	0.3609 $\pm$ 0.0140	0.3503 $\pm$ 0.0040	0.2838 $\pm$ 0.0042	0.5825 $\pm$ 0.0139	0.4724
	Magnitude-l1	0.7481 $\pm$ 0.0101	0.4530 $\pm$ 0.0050	0.3040 $\pm$ 0.0206	0.7239 $\pm$ 0.0092	0.3729 $\pm$ 0.0141	0.2703 $\pm$ 0.0037	0.2514 $\pm$ 0.0040	0.5596 $\pm$ 0.0140	0.4604
	Magnitude-l2	0.7225 $\pm$ 0.0104	0.4245 $\pm$ 0.0049	0.2380 $\pm$ 0.0191	0.6561 $\pm$ 0.0097	0.3038 $\pm$ 0.0134	0.2413 $\pm$ 0.0036	0.2258 $\pm$ 0.0041	0.5493 $\pm$ 0.0140	0.4202
	BI	0.6921 $\pm$ 0.0108	0.4272 $\pm$ 0.0049	0.2700 $\pm$ 0.0199	0.6511 $\pm$ 0.0098	0.3703 $\pm$ 0.0141	0.4968 $\pm$ 0.0040	0.3851 $\pm$ 0.0045	0.6661 $\pm$ 0.0133	0.4948
	Taylor	0.7002 $\pm$ 0.0107	0.4541 $\pm$ 0.0050	0.3020 $\pm$ 0.0206	0.6359 $\pm$ 0.0099	0.3695 $\pm$ 0.0141	0.5431 $\pm$ 0.0040	0.4048 $\pm$ 0.0045	0.6488 $\pm$ 0.0134	0.5073
Llama-3.1-8B-It	Dense	0.8003 $\pm$ 0.0093	0.5910 $\pm$ 0.0049	0.3380 $\pm$ 0.0212	0.8182 $\pm$ 0.0079	0.5179 $\pm$ 0.0146	0.6790 $\pm$ 0.0038	0.5552 $\pm$ 0.0045	0.7395 $\pm$ 0.0123	0.6299
	Reverse-order	0.7002 $\pm$ 0.0107	0.4010 $\pm$ 0.0049	0.2940 $\pm$ 0.0204	0.6170 $\pm$ 0.0100	0.3985 $\pm$ 0.0143	0.6342 $\pm$ 0.0039	0.5449 $\pm$ 0.0045	0.6243 $\pm$ 0.0136	0.5268
	Random	0.5653 $\pm$ 0.0116	0.2886 $\pm$ 0.0045	0.1400 $\pm$ 0.0155	0.3169 $\pm$ 0.0095	0.1860 $\pm$ 0.0114	0.2275 $\pm$ 0.0035	0.2559 $\pm$ 0.0041	0.5075 $\pm$ 0.0141	0.3110
	PPL	0.7628 $\pm$ 0.0099	0.4931 $\pm$ 0.0050	0.2640 $\pm$ 0.0197	0.7290 $\pm$ 0.0091	0.3805 $\pm$ 0.0142	0.3367 $\pm$ 0.0040	0.2724 $\pm$ 0.0041	0.5793 $\pm$ 0.0139	0.4772
	Magnitude-l1	0.5408 $\pm$ 0.0116	0.2634 $\pm$ 0.0044	0.1360 $\pm$ 0.0153	0.2845 $\pm$ 0.0093	0.2014 $\pm$ 0.0117	0.2504 $\pm$ 0.0037	0.2503 $\pm$ 0.0040	0.4878 $\pm$ 0.0140	0.3018
	Magnitude-l2	0.5413 $\pm$ 0.0116	0.2638 $\pm$ 0.0044	0.1340 $\pm$ 0.0152	0.2841 $\pm$ 0.0093	0.2014 $\pm$ 0.0117	0.2498 $\pm$ 0.0036	0.2504 $\pm$ 0.0040	0.4870 $\pm$ 0.0140	0.3015
	BI	0.7176 $\pm$ 0.0105	0.4196 $\pm$ 0.0049	0.2020 $\pm$ 0.0180	0.6107 $\pm$ 0.0100	0.2841 $\pm$ 0.0132	0.2417 $\pm$ 0.0036	0.2494 $\pm$ 0.0040	0.5391 $\pm$ 0.0140	0.4080
	Taylor	0.7138 $\pm$ 0.0105	0.4964 $\pm$ 0.0050	0.2740 $\pm$ 0.0200	0.6848 $\pm$ 0.0095	0.4181 $\pm$ 0.0144	0.2861 $\pm$ 0.0038	0.2504 $\pm$ 0.0040	0.7135 $\pm$ 0.0127	0.4796

4.1 Are fancy metrics essential for identifying redundant layers to prune?

The first question is to find the most “redundant” layers to prune. As discussed in Section3.1, there are various metrics for layer selection, which can be as straightforward as reverse-order, or as complicated as BI. However, does a complicated metric always contribute to a better performance? Probably not. We find that a simple metric, i.e., reverse-order, is competitive among these metrics.

Specifically, we conduct comprehensive experiments on Vicuna-7B-v1.5(Zheng etal., 2024), Qwen1.5-7B(Yang etal., 2024a), Gemma2-2B-Instruct(Team, 2024) and Llama-3.1-8B-Instruct(Dubey etal., 2024). We uniformly prune 8 layers (25% pruning ratio) for Vicuna-7B-v1.5, Qwen1.5-7B and Llama-3.1-8B-Instruct, and 6 layers for Gemma2-2B-Instruct. Experiments with a 50% pruning ratio (12 layers for Gemma2-2B-Instruct and 16 layers for others) are provided in TableA. In the fine-tuning stage, we use LoRA with a rank $d$ of $8$ and a batch size of $64$ , and the AdamW optimizer. The learning rate is set to $1\times 10^{-5}$ with $100$ warming steps.

Results. As shown in Table1, we find that the reverse-order metric delivers stable and superior results across various models under the 25% pruning rate, making it a reliable choice for pruning. On average, it outperforms the second-best PPL metric by $5.30\%$ across four models. The result also holds for the 50% pruning rate, as shown in TableA.We hope our insights can help researchers make informed choices when selecting the most suitable pruning metrics for their specific models.

4.2 Is the LoRA family the best choice for post-pruning fine-tuning?

In previous studies(Kim etal., 2024; Men etal., 2024), LoRA is often used to restore the performance of pruned models. This raises a question: Is the LoRA family the best choice for post-pruning fine-tuning? To answer this question, we further use QLoRA(Dettmers etal., 2024) and partial-layer fine-tuning techniques to conduct experiments. We briefly introduce these methods as follows:

LoRA Fine-tuning. LoRA is one of the best-performed parameter-efficient fine-tuning paradigm that updates dense model layers using pluggable low-rank matrices(Mao etal., 2024). Specifically, for a pre-trained weight matrix $W_{0}$ , LoRA constrains its update by representing the latter with a low-rank decomposition $W_{0}+\Delta W=W_{0}+BA$ . At the beginning of training, $A$ is initialize with a random Gaussian initialization, while $B$ is initialized to zero. During training, $W_{0}$ is frozen and does not receive gradient updates, while $A$ and $B$ contain trainable parameters. Then the forward pass can be formalized as:

W_{0}x+\Delta Wx=W_{0}x+BAx.

(4)

QLoRA Fine-tuning. QLoRA builds on LoRA by incorporating quantization techniques to further reduce memory usage while maintaining, or even enhancing the performance.

Partial-layer Fine-tuning. Compared to LoRA and QLoRA, which inject trainable low-rank factorization matrices into each layer, partial-layer fine-tuning simply freezes the weights of some layers while updating only the specified layers to save computing resources and time(Shen etal., 2021; Ngesthi etal., 2021; Peng & Wang, 2020). Following by the common practice of previous studies(Khan & Fang, 2023), we choose to fine-tune only the later layers that are closer to the output, while keeping the earlier layers, which capture more general features, frozen. Specifically, we use two different fine-tuning strategies: one is to finetune only the model head (lm_head only), and the other is to finetune the lm_head plus the last layer (lm_head + last layer), the last two layers (lm_head + last two layers), and the last three layers (lm_head + last three layers).

Model	Method	Layer	Benchmarks								Avg Acc
Model	Method	Layer	PIQA	HellaSwag	OpenbookQA	ARC-e	ARC-c	MMLU	CMMLU	WinoGrande	Avg Acc
Vicuna-7B-v1.5	LoRA	-	0.7171 $\pm$ 0.0105	0.5005 $\pm$ 0.0050	0.2608 $\pm$ 0.0198	0.6221 $\pm$ 0.0099	0.3848 $\pm$ 0.0142	0.4737 $\pm$ 0.0041	0.3417 $\pm$ 0.0044	0.6267 $\pm$ 0.0136	0.4909
	QLoRA	-	0.6649 $\pm$ 0.0110	0.4057 $\pm$ 0.0049	0.2700 $\pm$ 0.0199	0.5345 $\pm$ 0.0102	0.3439 $\pm$ 0.0139	0.4809 $\pm$ 0.0041	0.3473 $\pm$ 0.0044	0.6014 $\pm$ 0.0138	0.4561
	Partial-layer	lm_head only	0.7057 $\pm$ 0.0106	0.4865 $\pm$ 0.0050	0.2880 $\pm$ 0.0203	0.6301 $\pm$ 0.0099	0.4010 $\pm$ 0.0143	0.4819 $\pm$ 0.0041	0.3520 $\pm$ 0.0044	0.6156 $\pm$ 0.0137	0.4951
		lm_head+last layer	0.7155 $\pm$ 0.0105	0.5054 $\pm$ 0.0050	0.2900 $\pm$ 0.0203	0.6511 $\pm$ 0.0098	0.4113 $\pm$ 0.0144	0.4831 $\pm$ 0.0041	0.3538 $\pm$ 0.0044	0.6283 $\pm$ 0.0136	0.5048
		lm_head+last two layers	0.7214 $\pm$ 0.0105	0.5060 $\pm$ 0.0050	0.3020 $\pm$ 0.0206	0.6532 $\pm$ 0.0098	0.4002 $\pm$ 0.0143	0.4858 $\pm$ 0.0041	0.3530 $\pm$ 0.0044	0.6267 $\pm$ 0.0136	0.5060
		lm_head+last three layers	0.7247 $\pm$ 0.0104	0.5103 $\pm$ 0.0050	0.2960 $\pm$ 0.0204	0.6528 $\pm$ 0.0098	0.3985 $\pm$ 0.0143	0.4870 $\pm$ 0.0040	0.3544 $\pm$ 0.0044	0.6219 $\pm$ 0.0136	0.5057
Qwen1.5-7B	LoRA	-	0.6942 $\pm$ 0.0107	0.4444 $\pm$ 0.0050	0.2280 $\pm$ 0.0188	0.5143 $\pm$ 0.0103	0.3302 $\pm$ 0.0137	0.5101 $\pm$ 0.0041	0.7171 $\pm$ 0.0040	0.5912 $\pm$ 0.0138	0.5037
	QLoRA	-	0.6697 $\pm$ 0.0110	0.4028 $\pm$ 0.0049	0.2400 $\pm$ 0.0191	0.4760 $\pm$ 0.0102	0.2969 $\pm$ 0.0134	0.4797 $\pm$ 0.0041	0.6914 $\pm$ 0.0041	0.5825 $\pm$ 0.0139	0.4799
	Partial-layer	lm_head only	0.7149 $\pm$ 0.0105	0.4735 $\pm$ 0.0050	0.2460 $\pm$ 0.0193	0.5497 $\pm$ 0.0102	0.3524 $\pm$ 0.0140	0.5467 $\pm$ 0.0040	0.7276 $\pm$ 0.0039	0.5967 $\pm$ 0.0138	0.5259
		lm_head+last layer	0.7220 $\pm$ 0.0105	0.4850 $\pm$ 0.0050	0.2440 $\pm$ 0.0192	0.5690 $\pm$ 0.0102	0.3549 $\pm$ 0.0140	0.5719 $\pm$ 0.0040	0.7283 $\pm$ 0.0039	0.6275 $\pm$ 0.0136	0.5378
		lm_head+last two layers	0.7214 $\pm$ 0.0105	0.4915 $\pm$ 0.0050	0.2540 $\pm$ 0.0195	0.5783 $\pm$ 0.0101	0.3584 $\pm$ 0.0140	0.5734 $\pm$ 0.0040	0.7275 $\pm$ 0.0039	0.6298 $\pm$ 0.0136	0.5418
		lm_head+last three layers	0.7296 $\pm$ 0.0104	0.4974 $\pm$ 0.0050	0.2520 $\pm$ 0.0194	0.5808 $\pm$ 0.0101	0.3618 $\pm$ 0.0140	0.5795 $\pm$ 0.0040	0.7272 $\pm$ 0.0040	0.6275 $\pm$ 0.0136	0.5445
Llama-3.1-8B-It	LoRA	-	0.7002 $\pm$ 0.0107	0.4010 $\pm$ 0.0049	0.2940 $\pm$ 0.0204	0.6170 $\pm$ 0.0100	0.3985 $\pm$ 0.0143	0.6342 $\pm$ 0.0039	0.5449 $\pm$ 0.0045	0.6243 $\pm$ 0.0136	0.5268
	QLoRA	-	0.6980 $\pm$ 0.0107	0.3975 $\pm$ 0.0049	0.3000 $\pm$ 0.0205	0.6183 $\pm$ 0.0100	0.3840 $\pm$ 0.0142	0.6032 $\pm$ 0.0039	0.5090 $\pm$ 0.0045	0.6267 $\pm$ 0.0136	0.5171
	Partial-layer	lm_head only	0.7334 $\pm$ 0.0103	0.4896 $\pm$ 0.0050	0.2860 $\pm$ 0.0202	0.7012 $\pm$ 0.0094	0.4411 $\pm$ 0.0145	0.6122 $\pm$ 0.0040	0.5442 $\pm$ 0.0045	0.6717 $\pm$ 0.0132	0.5599
		lm_head+last layer	0.7350 $\pm$ 0.0103	0.5107 $\pm$ 0.0050	0.2940 $\pm$ 0.0204	0.7193 $\pm$ 0.0092	0.4531 $\pm$ 0.0145	0.6630 $\pm$ 0.0038	0.5526 $\pm$ 0.0045	0.6582 $\pm$ 0.0133	0.5732
		lm_head+last two layers	0.7361 $\pm$ 0.0103	0.5204 $\pm$ 0.0050	0.3080 $\pm$ 0.0207	0.7151 $\pm$ 0.0093	0.4633 $\pm$ 0.0146	0.6588 $\pm$ 0.0038	0.5543 $\pm$ 0.0045	0.6567 $\pm$ 0.0133	0.5766
		lm_head+last three layers	0.7383 $\pm$ 0.0103	0.5323 $\pm$ 0.0050	0.3080 $\pm$ 0.0207	0.7260 $\pm$ 0.0092	0.4684 $\pm$ 0.0146	0.6567 $\pm$ 0.0038	0.5515 $\pm$ 0.0045	0.6646 $\pm$ 0.0133	0.5807

Method	Benchmarks								Avg Acc
Method	PIQA	HellaSwag	OpenbookQA	ARC-e	ARC-c	MMLU	CMMLU	WinoGrande	Avg Acc
Dense	0.8003 $\pm$ 0.0093	0.5910 $\pm$ 0.0049	0.3380 $\pm$ 0.0212	0.8182 $\pm$ 0.0079	0.5179 $\pm$ 0.0146	0.6790 $\pm$ 0.0038	0.5552 $\pm$ 0.0045	0.7395 $\pm$ 0.0123	0.6299
lm_head+last three layers	0.7998 $\pm$ 0.0093	0.6057 $\pm$ 0.0049	0.3520 $\pm$ 0.0214	0.8186 $\pm$ 0.0079	0.5316 $\pm$ 0.0146	0.6784 $\pm$ 0.0038	0.5522 $\pm$ 0.0045	0.7316 $\pm$ 0.0125	0.6337
LoRA	0.8047 $\pm$ 0.0092	0.6007 $\pm$ 0.0049	0.3500 $\pm$ 0.0214	0.8287 $\pm$ 0.0077	0.5316 $\pm$ 0.0146	0.6764 $\pm$ 0.0038	0.5530 $\pm$ 0.0045	0.7380 $\pm$ 0.0124	0.6354

	LoRA	QLoRA	lm_head only	lm_head+last layer	lm_head+last two layers	lm_head+last three layers
Trainable parameters	15.73M	15.73M	525.34M	743.45M	961.56M	1179.68M
GPU memory	45.83G	14.26G	39.82G	42.12G	44.41G	48.02G
Training time (2 epoch)	10440.30s	17249.01s	6952.92s	7296.76s	7616.83s	7931.36s

In view of the superiority of the reverse-order metric in Section4.1, we use it to prune here. For the Vicuna-7B-v1.5, Qwen1.5-7B, and Llama-3.1-8B-Instruct models, we prune 8 layers. For the Gemma2-2B-Instruct model, we prune 6 layers. Subsequently, we utilize LoRA, QLoRA and partial-layer fine-tuning methods to restore performance. We provide more results of fine-tuning with the taylor metric in TableB. In particular, because Gemma2-2B-Instruct employs weight tying(Press & Wolf, 2016) to share the weights between the embedding layer and the softmax layer (lm_head), we exclude partial-layer fine-tuning in Gemma2-2B-Instruct. For fine-tuning with LoRA and partial-layer methods, we utilize the AdamW optimizer, while for QLoRA, we opt for the paged_adamw_8bit optimizer. All other hyperparameter settings are the same as in Section4.1.

Results. As shown in the Table2 and TableB, we find that fine-tuning with QLoRA slightly hurts the performance of pruned models compared to LoRA. Excitingly, the effect of partial-layer fine-tuning is significantly better than LoRA, providing a viable new direction for fine-tuning models after pruning. In the ablation study, we compare the performance of LoRA with partial-layer fine-tuning for the full model in Table3, which shows that partial-layer fine-tuning and LoRA perform similarly. This suggests that the conventional insights for the full model fine-tuning do not hold after pruning, i.e., the structural changes and parameter reduction of the model enable partial layer fine-tuning to adapt more effectively to the new parameter distribution and fully leverage the potential benefits of pruning. When considering fine-tuning methods for LLMs, in addition to performance, the training cost is also a significant factor to take into account. Therefore, we compare the training cost of these fine-tuning methods, including training time, gpu memory and trainable parameters. Specifically, we conduct experiments on 2 empty NVIDIA RTX A100 GPUs using the pruned Llama-3.1-8B-Instruct model (with 8 layers removed in reverse order). Table4 shows the comparison among these fine-tuning methods. We find that compared to LoRA, partial-layer fine-tuning involves more trainable parameters but maintains comparable GPU usage and achieves faster training time. Additionally, partial-layer fine-tuning outperforms LoRA in effectiveness. In contrast, although QLoRA consumes less GPU memory, it has much longer training time and yields poorer performance. In summary, we conclude that partial-layer fine-tuning is an effective approach to restoring the performance of pruned models when sufficient memory is available.

4.3 Will iterative pruning outperform one-shot pruning?

In this subsection, we provide insights into the optimal pruning strategy for LLMs. Although Muralidharan etal. (2024) have explored pruning strategies and concluded that iterative pruning offers no benefit, their study focuses on utilizing knowledge distillation(Hinton, 2015) for performance recovery. In contrast, this paper concentrates on layer pruning with LoRA and partial-layer fine-tuning, thereby broadening the scope of pruning strategies evaluated. We briefly introduce the one-shot pruning and iterative pruning:

One-shot Pruning.One-shot pruning scores once and then prune the model to a target prune ratio.

Iterative Pruning.Iterative pruning alternately processes the score-prune-update cycle until achieving the target prune ratio.

Specifically, we select Llama-3.1-8B-Instruct and Gemma2-2B-Instruct as the base models. For one-shot pruning, we prune 8 layers from the Llama-3.1-8B-Instruct and 6 layers from the Gemma2-2B-Instruct in a single step, guided by the reverse-order and taylor metrics. For iterative pruning with LoRA, we begin by scoring all layers using these metrics. Subsequently, we set the pruning step to $1$ and $4$ for Llama-3.1-8B-Instruct, and $1$ and $3$ for Gemma2-2B-Instruct. After each pruning step, we fine-tune the model with LoRA and merge LoRA weights back into the fine-tuned model. This score-prune-fine-tune-merge cycle is repeated until a total of 8 layers are pruned for Llama-3.1-8B-Instruct and 6 layers for Gemma2-2B-Instruct. For iterative pruning with partial-layer fine-tuning, we fine-tune the model using partial-layer fine-tuning (lm_head + last three layers) after each pruning step, and then repeat the score-prune-fine-tune cycle. To avoid the fine-tuned layers being pruned completely, we set the pruning step size to 1. All hyperparameter settings are the same as in Section4.1. Experiments with iterative pruning of more layers are provided in TableC.

Results. By comparing the results of iterative and one-shot pruning in Table5 and TableC, we find that unlike traditional CNN pruning, which often yields significant performance improvements through iterative pruning(Tan & Motani, 2020; He & Xiao, 2023), the iterative approach for LLMs may not provide the same benefits and can even lead to performance degradation. We believe that is because too much training causes the model to suffer from catastrophic forgetting(Zhai etal., 2024; Liu etal., 2024a). Figure B visualizes the representational similarity of different pruning strategies. From this, we observe that different pruning strategies yield significantly different representations, highlighting the impact of each strategy on the model’s learned features. Besides, iterative pruning requires more computational overhead than one-shot pruning, which is not cost-effective with limited performance gains.

Fine-tuning Method

Model

Metric

Iteration steps

Benchmarks

Avg Acc

PIQA

HellaSwag

OpenbookQA

ARC-e

ARC-c

MMLU

CMMLU

WinoGrande

LoRA

Llama-3.1-8B-It

Reverse-order

one-shot

0.7002+0.0107

0.4010+0.0049

0.2940+0.0204

0.6170+0.0100

0.3985+0.0143

0.6342+0.0039

0.5449

\pm

0.0045

0.6243

\pm

0.0136

0.5268

1:4:8

0.7176

\pm

0.0105

0.4538

\pm

0.0050

0.2920

\pm

0.0204

0.6705

\pm

0.0096

0.4121

\pm

0.0144

0.6374

\pm

0.0039

0.5439

\pm

0.0045

0.6369

\pm

0.0135

0.5455

1:1:8

0.7160

\pm

0.0105

0.4470

\pm

0.0050

0.2860

\pm

0.0202

0.6637

\pm

0.0097

0.4061

\pm

0.0144

0.6440

\pm

0.0039

0.5425

\pm

0.0045

0.6448

\pm

0.0135

0.5438

Taylor

one-shot

0.7138

\pm

0.0105

0.4964

\pm

0.0050

0.2740

\pm

0.0200

0.6848

\pm

0.0095

0.4181

\pm

0.0144

0.2861

\pm

0.0038

0.2504

\pm

0.0040

0.7135

\pm

0.0127

0.4796

1:4:8

0.7149

\pm

0.0105

0.4991

\pm

0.0050

0.2480

\pm

0.0193

0.7071

\pm

0.0093

0.3951

\pm

0.0143

0.4676

\pm

0.0041

0.3480

\pm

0.0044

0.6709

\pm

0.0132

0.5063

1:1:8

0.6921

\pm

0.0108

0.4728

\pm

0.0050

0.2140

\pm

0.0184

0.6675

\pm

0.0097

0.3891

\pm

0.0142

0.4576

\pm

0.0041

0.3511

\pm

0.0044

0.6519

\pm

0.0134

0.4870

Gemma2-2B-It

Reverse-order

one-shot

0.7029

\pm

0.0107

0.4529

\pm

0.0050

0.2660

\pm

0.0198

0.6343

\pm

0.0099

0.3763

\pm

0.0142

0.5261

\pm

0.0040

0.4117

\pm

0.0045

0.6551

\pm

0.0134

0.5032

1:3:6

0.6953

\pm

0.0107

0.4523

\pm

0.0050

0.2900

\pm

0.0203

0.6397

\pm

0.0099

0.3729

\pm

0.0141

0.5418

\pm

0.0040

0.4013

\pm

0.0045

0.6496

\pm

0.0134

0.5054

1:1:6

0.7067

\pm

0.0106

0.4476

\pm

0.0050

0.2660

\pm

0.0198

0.6305

\pm

0.0099

0.3746

\pm

0.0141

0.5143

\pm

0.0040

0.4066

\pm

0.0045

0.6559

\pm

0.0134

0.5003

Taylor

one-shot

0.7002

\pm

0.0107

0.4541

\pm

0.0050

0.3020

\pm

0.0206

0.6359

\pm

0.0099

0.3695

\pm

0.0141

0.5431

\pm

0.0040

0.4048

\pm

0.0045

0.6488

\pm

0.0134

0.5073

1:3:6

0.7057

\pm

0.0106

0.4473

\pm

0.0050

0.2380

\pm

0.0191

0.6553

\pm

0.0098

0.3490

\pm

0.0139

0.3697

\pm

0.0040

0.2884

\pm

0.0042

0.5927

\pm

0.0138

0.4558

1:1:6

0.7236

\pm

0.0104

0.4544

\pm

0.0050

0.2860

\pm

0.0202

0.6574

\pm

0.0097

0.3490

\pm

0.0139

0.4763

\pm

0.0041

0.3801

\pm

0.0045

0.6306

\pm

0.0136

0.4947

Partial-layer

Llama-3.1-8B-It

Reverse-order

one-shot

0.7383

\pm

0.0103

0.5323

\pm

0.0050

0.3080

\pm

0.0207

0.7260

\pm

0.0092

0.4684

\pm

0.0146

0.6567

\pm

0.0038

0.5515

\pm

0.0045

0.6646

\pm

0.0133

0.5807

1:1:8

0.7432

\pm

0.0102

0.5357

\pm

0.0050

0.2980

\pm

0.0205

0.7496

\pm

0.0089

0.4590

\pm

0.0146

0.6539

\pm

0.0038

0.5558

\pm

0.0045

0.6922

\pm

0.0130

0.5859

Taylor

one-shot

0.7345

\pm

0.0103

0.5290

\pm

0.0050

0.3020

\pm

0.0206

0.7399

\pm

0.0090

0.4360

\pm

0.0145

0.6277

\pm

0.0039

0.4763

\pm

0.0046

0.7151

\pm

0.0127

0.5701

1:1:8

0.6300

\pm

0.0113

0.3553

\pm

0.0048

0.1760

\pm

0.0170

0.5177

\pm

0.0103

0.2756

\pm

0.0131

0.2611

\pm

0.0037

0.2557

\pm

0.0041

0.5312

\pm

0.0140

0.3753

	Verification	PPL on WikiText2		PPL on PTB		Avg Acc
	Metric	BI	Taylor	BI	Taylor	BI	Taylor
Calibration Samples	1	51.06	65.43	90.97	94.35	0.40	0.36
	5	43.54	65.43	79.34	94.35	0.43	0.36
	10	53.53	65.43	101.64	94.35	0.41	0.36
	30	50.03	55.42	88.02	77.63	0.42	0.55
	50	59.73	55.42	103.19	77.63	0.41	0.55

5 Sensitivity Analysis

In this section, we conduct sensitivity analyses on the number of calibration samples, the choice of SFT dataset and various pruning rates for LLM layer pruning.

The effect of number of calibration samples on LLM layer pruning.It is worth noting that some data-driven layer pruning methods, such as BI and Taylor, rely upon calibration samples to generate layer activations. Therefore, we explore the effect of the number of calibration samples on pruning. Specifically, we calculate BI and Taylor metrics using 1, 5, 10, 30, and 50 calibration samples, prune 8 layers based on these metrics, finetune the pruned Llama-3.1-8B-Instruct models using LoRA, and evaluate their performance through lm-evaluation-harness package. For ease of comparison, we report the average accuracy on 8 datasets in the main text. For more details, see TableD. Besides, we report the model perplexity on the WikiText and Penn Treebank test set. As shown in Table6, we observe that the pruned models, obtained using varying numbers of calibration samples, do affect the model complexity and zero-shot performance, which suggests that for data-driven pruning methods, performance stability should also be considered a key criterion when evaluating the quality of pruning technique.

Dataset	Benchmarks								Avg Acc
Dataset	PIQA	HellaSwag	OpenbookQA	ARC-e	ARC-c	MMLU	CMMLU	WinoGrande	Avg Acc
Dolly-15k	0.7709 $\pm$ 0.0098	0.5541 $\pm$ 0.0050	0.3000 $\pm$ 0.0205	0.7424 $\pm$ 0.0090	0.4838 $\pm$ 0.0146	0.6753 $\pm$ 0.0038	0.5522 $\pm$ 0.0045	0.7032 $\pm$ 0.0128	0.5977
Alpaca-cleaned	0.7383 $\pm$ 0.0103	0.5323 $\pm$ 0.0050	0.3080 $\pm$ 0.0207	0.7260 $\pm$ 0.0092	0.4684 $\pm$ 0.0146	0.6567 $\pm$ 0.0038	0.5515 $\pm$ 0.0045	0.6646 $\pm$ 0.0133	0.5807
MMLU	0.6012 $\pm$ 0.0114	0.2714 $\pm$ 0.0044	0.1700 $\pm$ 0.0168	0.3430 $\pm$ 0.0097	0.2457 $\pm$ 0.0126	0.5888 $\pm$ 0.0040	0.5266 $\pm$ 0.0045	0.5856 $\pm$ 0.0138	0.4165

Reassessing Layer Pruning in LLMs: New Insights and Methods (3)

Reassessing Layer Pruning in LLMs: New Insights and Methods (4)

The effect of SFT datasets on LLM layer pruning. In the previous sections, we uniformly utilize Alpaca-cleaned(Taori etal., 2023) to fine-tune the pruned models. Herein, we aim to assess how fine-tuning a pruned model using different SFT datasets affects its performance. Specifically, we conduct experiments using the Reverse-order metric to remove 8 layers from the Llama-3.1-8B-Instruct and fine-tune the pruned model using lm_head + last three layers on MMLU (training set)(Hendrycks etal., 2021) and Dolly-15k(Conover etal., 2023). We set the maximum sequence length to 512 for MMLU and 1024 for Dolly-15k. From Table7, we observe that among these datasets, Dolly-15k achieves the best results, followed by Alpaca-cleaned. This demonstrates that fine-tuning with different SFT datasets has a significant impact on the performance of pruned models and suggests further exploration of the most suitable datasets for fine-tuning pruned models.

The effect of different pruning rates on LLM layer pruning.We investigate the impact of pruning the LLM at various pruning rates in Figure2. Specifically, we conduct one-shot pruning on Llama-3.1-8B-Instruct using reverse-order and taylor metrics and evaluate their effects on the model’s performance with LoRA. All hyperparameter settings remain consistent with those in Section4.1. As shown in Figure2, we observe that as the number of pruned layers increases, the performance of the model on all datasets tends to decrease and eventually converges.However, certain datasets, especially MMLU, CMMLU, and ARC-c, are highly sensitive to layer changes and degrade faster than others. Besides, after cutting off about 16 layers, the model was damaged, so we set the maximum pruning rate in the paper to 16 layers.

Baseline	# Parameters (TTokens)	Benchmarks								Avg Acc
Baseline	# Parameters (TTokens)	PIQA	HellaSwag	OpenbookQA	ARC-e	ARC-c	MMLU	CMMLU	WinoGrande	Avg Acc
Vicuna-7B-v1.5	6.74B (370M)	0.7720 $\pm$ 0.0098	0.5642 $\pm$ 0.0049	0.3300 $\pm$ 0.0210	0.7555 $\pm$ 0.0088	0.4326 $\pm$ 0.0145	0.4858 $\pm$ 0.0040	0.3518 $\pm$ 0.0044	0.6953 $\pm$ 0.0129	0.5484
ChatGLM2-6B	6.24B (1.4T)	0.5403 $\pm$ 0.0116	0.2589 $\pm$ 0.0044	0.1420 $\pm$ 0.0156	0.2597 $\pm$ 0.0090	0.2005 $\pm$ 0.0117	0.2431 $\pm$ 0.0036	0.2537 $\pm$ 0.0040	0.5288 $\pm$ 0.0140	0.3034
Baichuan2-7B	7.51B (2.6T)	0.7666 $\pm$ 0.0099	0.5363 $\pm$ 0.0050	0.3020 $\pm$ 0.0206	0.7475 $\pm$ 0.0089	0.4206 $\pm$ 0.0144	0.5024 $\pm$ 0.0040	0.5220 $\pm$ 0.0045	0.6819 $\pm$ 0.0131	0.5599
Qwen1.5-7B	7.72B (18T)	0.7845 $\pm$ 0.0096	0.5785 $\pm$ 0.0049	0.3160 $\pm$ 0.0208	0.7125 $\pm$ 0.0093	0.4053 $\pm$ 0.0143	0.5967 $\pm$ 0.0039	0.7277 $\pm$ 0.0039	0.6575 $\pm$ 0.0133	0.5973
LLaMA3-8B	8.03B (15T+)	0.7965 $\pm$ 0.0094	0.6014 $\pm$ 0.0049	0.3480 $\pm$ 0.0213	0.8005 $\pm$ 0.0082	0.4983 $\pm$ 0.0146	0.6212 $\pm$ 0.0038	0.4752 $\pm$ 0.0045	0.7332 $\pm$ 0.0124	0.6093
Gemma2-7B	8.54B (6T)	0.8025 $\pm$ 0.0093	0.6039 $\pm$ 0.0049	0.3300 $\pm$ 0.0210	0.8110 $\pm$ 0.0080	0.5009 $\pm$ 0.0146	0.6143 $\pm$ 0.0039	0.4430 $\pm$ 0.0045	0.7435 $\pm$ 0.0123	0.6061
Llama-3.1-8B-It	8.03B (15T+)	0.8003 $\pm$ 0.0093	0.5910 $\pm$ 0.0049	0.3380 $\pm$ 0.0212	0.8182 $\pm$ 0.0079	0.5179 $\pm$ 0.0146	0.6790 $\pm$ 0.0038	0.5552 $\pm$ 0.0045	0.7395 $\pm$ 0.0123	0.6299
ShortGPT (BI)	6.29B (12.74M)	0.7176 $\pm$ 0.0105	0.4196 $\pm$ 0.0049	0.2020 $\pm$ 0.0180	0.6107 $\pm$ 0.0100	0.2841 $\pm$ 0.0132	0.2417 $\pm$ 0.0036	0.2494 $\pm$ 0.0040	0.5391 $\pm$ 0.0140	0.4080
Shortened LLaMA (PPL)	6.29B (12.74M)	0.7628 $\pm$ 0.0099	0.4931 $\pm$ 0.0050	0.2640 $\pm$ 0.0197	0.7290 $\pm$ 0.0091	0.3805 $\pm$ 0.0142	0.3367 $\pm$ 0.0040	0.2724 $\pm$ 0.0041	0.5793 $\pm$ 0.0139	0.4772
Shortened LLaMA (Taylor)	6.29B (12.74M)	0.7138 $\pm$ 0.0105	0.4964 $\pm$ 0.0050	0.2740 $\pm$ 0.0200	0.6848 $\pm$ 0.0095	0.4181 $\pm$ 0.0144	0.2861 $\pm$ 0.0038	0.2504 $\pm$ 0.0040	0.7135 $\pm$ 0.0127	0.4796
Llama-3.1-6.3B-It-Alpaca	6.29B (12.74M)	0.7383 $\pm$ 0.0103	0.5323 $\pm$ 0.0050	0.3080 $\pm$ 0.0207	0.7260 $\pm$ 0.0092	0.4684 $\pm$ 0.0146	0.6567 $\pm$ 0.0038	0.5515 $\pm$ 0.0045	0.6646 $\pm$ 0.0133	0.5807
Llama-3.1-6.3B-It-Dolly	6.29B (14.96M)	0.7709 $\pm$ 0.0098	0.5541 $\pm$ 0.0050	0.3000 $\pm$ 0.0205	0.7424 $\pm$ 0.0090	0.4838 $\pm$ 0.0146	0.6753 $\pm$ 0.0038	0.5522 $\pm$ 0.0045	0.7032 $\pm$ 0.0128	0.5977

Model	# Params	# MACs	Memory	Latency
Llama-3.1-6.3B-It-Alpaca, Llama-3.1-6.3B-Dolly	6.29B	368.65G	23984MiB	210.35s

6 Obtaining the Best Pruned Models

In Section4 and Section5, we have gained some valuable non-trivial practices and insights on LLM layer pruning through systematic experiments. Herein, we use these practices and insights to obtain the Llama-3.1-6.3B-It model and compare its performance against multiple baselines: (1) the original Llama-3.1-8B-It model, (2) a set of similarly sized community models and (3) a set of pruned models obtained by state-of-the-art LLM layer pruning methods (all prune 8 layers, fine-tune on Alpaca-cleaned).

Specifically, Llama-3.1-6.3B-It is obtained by pruning 8 layers of Llama-3.1-8B-It using the reverse-order metric. Note that, in contrast to these community models trained from scratch on trillions of tokens (except for Vicuna-7B-v1.5), Llama-3.1-6.3B-It is fine-tuned solely on Alpaca-cleaned (12.74M tokens) and Dolly-15k (14.96M tokens). For ease of distinction, we refer to them as “Llama-3.1-6.3B-It-Alpaca” and “Llama-3.1-6.3B-It-Dolly”, respectively. From Table8, we find that both Llama-3.1-6.3B-It-Alpaca and Llama-3.1-6.3B-It-Dolly outperform ChatGLM2-6B(GLM etal., 2024), Vicuna-7B-v1.5(Zheng etal., 2024) and Baichuan2-7B(Baichuan, 2023), and partially exceed LLaMA3-8B(AI@Meta, 2024), Gemma2-7B(Team etal., 2024) (e.g., MMLU), while using significantly fewer training tokens. Notably, Llama-3.1-6.3B-It-Dolly also outperforms Qwen1.5-7B(Yang etal., 2024a). Besides, we also compare our models to other pruned models obtained by various LLM layer pruning methods. Experimental results show that our models are nearly 19% better than ShortGPT(Men etal., 2024) and 10%+ better than Shortened LLaMA(Kim etal., 2024). Table9 presents the statistic of Llama-3.1-6.3B-It, including parameters, MACs, memory requirements and latency. Following Ma etal. (2023a), the statistical evaluation is conducted in inference mode, where the model is fed a sentence consisting of 64 tokens. The latency is tested under the test set of WikiText2 on a single NVIDIA RTX A100 GPU. We also present the generation results of the Llama-3.1-6.3B-It-Alpaca, Llama-3.1-6.3B-It-Dolly and Llama-3.1-8B-It in TableE.

7 Conclusion

In this paper, we revisit LLM layer pruning, focusing on pruning metrics, fine-tuning methods and pruning strategies. From these efforts, we have developed a practical list of best practices for LLM layer pruning. We use these practices and insights to guide the pruning of Llama-3.1-8B-Instruct and obtain Llama-3.1-6.3B-It-Alpaca and Llama-3.1-6.3B-It-Dolly. Our pruned models require fewer training tokens compared to training from scratch, yet still performing favorably against various popular community LLMs of similar size. We hope our work will help inform best practices for deploying LLMs in real-world applications.

Limitations and Future Work. In Section5, we find that SFT datasets do effect the performance of pruned models. Therefore, we will explore which SFT datasets are more suitable for fine-tuning pruned models in future work. Additionally, in this paper, we focus primarily on layer pruning due to the straightforward nature of pruning layers in LLMs, where the input and output dimensions are identical. However, we plan to further investigate weight pruning(Sun etal., 2023; Frantar & Alistarh, 2023) and width pruning(Xia etal., 2023; Ma etal., 2023b) in future experiments.

8 Reproducibility Statement

The authors have made great efforts to ensure the reproducibility of the empirical results reported in this paper. Firstly, the experiment settings, evaluation metrics, and datasets were described in detail in Section3.2. Secondly, the code to reproduce the results is available at https://github.com/yaolu-zjut/Navigation-LLM-layer-pruning, and the optimal model weights can be found at at https://huggingface.co/YaoLuzjut/Llama-3.1-6.3B-It-Alpaca and https://huggingface.co/YaoLuzjut/Llama-3.1-6.3B-It-Dolly.

9 Ethics statement

In this paper, we carefully consider ethical concerns related to our research and ensure that all methodologies and experimental designs adhere to ethical standards. Our study focuses on layer pruning to enhance the efficiency of LLMs and reduce computational resource requirements, thereby promoting sustainable AI development. Furthermore, all models and datasets used in our research are sourced from publicly available and accessible origins, ensuring no infringement on intellectual property or personal privacy.

References

Achiam etal. (2023)Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, FlorenciaLeoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, etal.Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023.
AI@Meta (2024)AI@Meta.Llama 3 model card.2024.URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.
Baichuan (2023)Baichuan.Baichuan 2: Open large-scale language models.arXiv preprint arXiv:2309.10305, 2023.URL https://arxiv.org/abs/2309.10305.
Bisk etal. (2020)Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, etal.Piqa: Reasoning about physical commonsense in natural language.In Proceedings of the AAAI conference on artificial intelligence, volume34, pp. 7432–7439, 2020.
Chen & Zhao (2018)Shi Chen and QiZhao.Shallowing deep networks: Layer-wise pruning based on feature representations.IEEE transactions on pattern analysis and machine intelligence, 41(12):3048–3056, 2018.
Chen etal. (2024)Xiaodong Chen, Yuxuan Hu, and Jing Zhang.Compressing large language models by streamlining the unimportant layer.arXiv preprint arXiv:2403.19135, 2024.
Clark etal. (2018)Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018.
Conover etal. (2023)Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin.Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023.URL https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm.
Deng etal. (2023)Xiang Deng, Vasilisa Bashlovkina, Feng Han, Simon Baumgartner, and Michael Bendersky.Llms to the moon? reddit market sentiment analysis with large language models.In Companion Proceedings of the ACM Web Conference 2023, pp. 1014–1019, 2023.
Dettmers etal. (2024)Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.Qlora: Efficient finetuning of quantized llms.Advances in Neural Information Processing Systems, 36, 2024.
Dosovitskiy etal. (2021)Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.An image is worth 16x16 words: Transformers for image recognition at scale, 2021.URL https://arxiv.org/abs/2010.11929.
Dubey etal. (2024)Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, etal.The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024.
Frantar & Alistarh (2023)Elias Frantar and Dan Alistarh.Sparsegpt: Massive language models can be accurately pruned in one-shot.In International Conference on Machine Learning, pp. 10323–10337. PMLR, 2023.
Gao etal. (2023)Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain LeNoac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.A framework for few-shot language model evaluation, 12 2023.URL https://zenodo.org/records/10256836.
GLM etal. (2024)Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, DaYin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, WengLam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang.Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024.
Gu etal. (2024)Yuxian Gu, LiDong, Furu Wei, and Minlie Huang.Minillm: Knowledge distillation of large language models.In The Twelfth International Conference on Learning Representations, 2024.
Guenter & Sideris (2024)Valentin FrankIngmar Guenter and Athanasios Sideris.Concurrent training and layer pruning of deep neural networks.arXiv preprint arXiv:2406.04549, 2024.
He etal. (2015)Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.Deep residual learning for image recognition, 2015.URL https://arxiv.org/abs/1512.03385.
He & Xiao (2023)Yang He and Lingao Xiao.Structured pruning for deep convolutional neural networks: A survey.IEEE transactions on pattern analysis and machine intelligence, 2023.
Hendrycks etal. (2021)Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Hinton (2015)Geoffrey Hinton.Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015.
Hu etal. (2021)EdwardJ Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, LuWang, and Weizhu Chen.Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021.
Jaiswal etal. (2023)Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, and Yinfei Yang.Compressing llms: The truth is rarely pure and never simple.arXiv preprint arXiv:2310.01382, 2023.
Khan & Fang (2023)MuhammadOsama Khan and YiFang.Revisiting fine-tuning strategies for self-supervised medical imaging analysis.arXiv preprint arXiv:2307.10915, 2023.
Kim etal. (2024)Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song.Shortened llama: A simple depth pruning for large language models.arXiv preprint arXiv:2402.02834, 2024.
Lee etal. (2024)Jungi Lee, Wonbeom Lee, and Jaewoong Sim.Tender: Accelerating large language models via tensor decomposition and runtime requantization.arXiv preprint arXiv:2406.12930, 2024.
Li etal. (2016)Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and HansPeter Graf.Pruning filters for efficient convnets.arXiv preprint arXiv:1608.08710, 2016.
Li etal. (2023)Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin.Cmmlu: Measuring massive multitask language understanding in chinese.arXiv preprint arXiv:2306.09212, 2023.
Lin etal. (2024)JiLin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han.Awq: Activation-aware weight quantization for on-device llm compression and acceleration.Proceedings of Machine Learning and Systems, 6:87–100, 2024.
Liu etal. (2024a)Chengyuan Liu, Shihang Wang, Yangyang Kang, Lizhi Qing, Fubang Zhao, Changlong Sun, Kun Kuang, and Fei Wu.More than catastrophic forgetting: Integrating general capabilities for domain-specific llms.arXiv preprint arXiv:2405.17830, 2024a.
Liu etal. (2024b)Deyuan Liu, Zhanyue Qin, Hairu Wang, Zhao Yang, Zecheng Wang, Fangying Rong, Qingbin Liu, Yanchao Hao, XiChen, Cunhang Fan, etal.Pruning via merging: Compressing llms via manifold alignment based layer merging.arXiv preprint arXiv:2406.16330, 2024b.
Liu etal. (2024c)Songwei Liu, Chao Zeng, Lianqiang Li, Chenqian Yan, Lean Fu, Xing Mei, and Fangmin Chen.Foldgpt: Simple and effective large language model compression scheme.arXiv preprint arXiv:2407.00928, 2024c.
Liu etal. (2021)ZeLiu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.Swin transformer: Hierarchical vision transformer using shifted windows, 2021.URL https://arxiv.org/abs/2103.14030.
Liu etal. (2023)Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra.Llm-qat: Data-free quantization aware training for large language models.arXiv preprint arXiv:2305.17888, 2023.
Lu etal. (2022)Yao Lu, Wen Yang, Yunzhe Zhang, Zuohui Chen, Jinyin Chen, QiXuan, Zhen Wang, and Xiaoniu Yang.Understanding the dynamics of dnns using graph modularity.In European Conference on Computer Vision, pp. 225–242. Springer, 2022.
Ma etal. (2023a)Xinyin Ma, Gongfan Fang, and Xinchao Wang.Llm-pruner: On the structural pruning of large language models.Advances in neural information processing systems, 36:21702–21720, 2023a.
Ma etal. (2023b)Xinyin Ma, Gongfan Fang, and Xinchao Wang.Llm-pruner: On the structural pruning of large language models.In Advances in Neural Information Processing Systems, 2023b.
Mao etal. (2024)Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, YuMi, Zhonghao Hu, and Yunjun Gao.A survey on lora of large language models.arXiv preprint arXiv:2407.11046, 2024.
Marcus etal. (1993)Mitch Marcus, Beatrice Santorini, and MaryAnn Marcinkiewicz.Building a large annotated corpus of english: The penn treebank.Computational linguistics, 19(2):313–330, 1993.
Men etal. (2024)Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen.Shortgpt: Layers in large language models are more redundant than you expect.arXiv preprint arXiv:2403.03853, 2024.
Meng etal. (2024)Fanxu Meng, Zhaohui Wang, and Muhan Zhang.Pissa: Principal singular values and singular vectors adaptation of large language models, 2024.URL https://arxiv.org/abs/2404.02948.
Merity etal. (2016)Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843, 2016.
Mihaylov etal. (2018)Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018.
Muralidharan etal. (2024)Saurav Muralidharan, SharathTuruvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov.Compact language models via pruning and knowledge distillation.arXiv preprint arXiv:2407.14679, 2024.
Ngesthi etal. (2021)StephanyOctaviani Ngesthi, Iwan Setyawan, and IvannaK Timotius.The effect of partial fine tuning on alexnet for skin lesions classification.In 2021 13th International Conference on Information Technology and Electrical Engineering (ICITEE), pp. 147–152. IEEE, 2021.
Peng & Wang (2020)Peng Peng and Jiugen Wang.How to fine-tune deep neural networks in few-shot learning?arXiv preprint arXiv:2012.00204, 2020.
Press & Wolf (2016)Ofir Press and Lior Wolf.Using the output embedding to improve language models.arXiv preprint arXiv:1608.05859, 2016.
Saha etal. (2023)Rajarshi Saha, Varun Srivastava, and Mert Pilanci.Matrix compression via randomized low rank and low precision factorization.Advances in Neural Information Processing Systems, 36, 2023.
Sakaguchi etal. (2021)Keisuke Sakaguchi, RonanLe Bras, Chandra Bhagavatula, and Yejin Choi.Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021.
Shah etal. (2024)Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao.Flashattention-3: Fast and accurate attention with asynchrony and low-precision.arXiv preprint arXiv:2407.08608, 2024.
Shen etal. (2021)Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang-Ting Cheng.Partial is better than all: Revisiting fine-tuning strategy for few-shot learning.In Proceedings of the AAAI conference on artificial intelligence, volume35, pp. 9594–9602, 2021.
Siddiqui etal. (2024)ShoaibAhmed Siddiqui, Xin Dong, Greg Heinrich, Thomas Breuel, Jan Kautz, David Krueger, and Pavlo Molchanov.A deeper look at depth pruning of llms.arXiv preprint arXiv:2407.16286, 2024.
Simonyan & Zisserman (2015)Karen Simonyan and Andrew Zisserman.Very deep convolutional networks for large-scale image recognition, 2015.URL https://arxiv.org/abs/1409.1556.
Sun etal. (2023)Mingjie Sun, Zhuang Liu, Anna Bair, and JZico Kolter.A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695, 2023.
Szegedy etal. (2014)Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.Going deeper with convolutions, 2014.URL https://arxiv.org/abs/1409.4842.
Tan & Motani (2020)Chong MinJohn Tan and Mehul Motani.Dropnet: Reducing neural network complexity via iterative pruning.In International Conference on Machine Learning, pp. 9356–9366. PMLR, 2020.
Tang etal. (2023)Hui Tang, Yao Lu, and QiXuan.Sr-init: An interpretable layer pruning method.In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE, 2023.
Taori etal. (2023)Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and TatsunoriB Hashimoto.Stanford alpaca: An instruction-following llama model, 2023.
Team (2024)Gemma Team.Gemma.2024.doi: 10.34740/KAGGLE/M/3301.URL https://www.kaggle.com/m/3301.
Team etal. (2024)Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, MihirSanjay Kale, Juliette Love, etal.Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024.
Tishby etal. (2000)Naftali Tishby, FernandoC Pereira, and William Bialek.The information bottleneck method.arXiv preprint physics/0004057, 2000.
Touvron etal. (2023)Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, etal.Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023.
Wang etal. (2023)Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu.Document-level machine translation with large language models.arXiv preprint arXiv:2304.02210, 2023.
Wang etal. (2019)Wenxiao Wang, Shuai Zhao, Minghao Chen, Jinming Hu, Deng Cai, and Haifeng Liu.Dbp: Discrimination based block-level pruning for deep model acceleration.arXiv preprint arXiv:1912.10178, 2019.
Williams & Aletras (2024)Miles Williams and Nikolaos Aletras.On the impact of calibration data in post-training quantization and pruning.In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10100–10118, 2024.
Xia etal. (2023)Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen.Sheared llama: Accelerating language model pre-training via structured pruning.arXiv preprint arXiv:2310.06694, 2023.
Xu etal. (2024)Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou.A survey on knowledge distillation of large language models.arXiv preprint arXiv:2402.13116, 2024.
Yang etal. (2024a)AnYang, Baosong Yang, Binyuan Hui, BoZheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, NaNi, Pei Zhang, Peng Wang, RuPeng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, YuWan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan.Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024a.
Yang etal. (2024b)Yifei Yang, Zouying Cao, and Hai Zhao.Laco: Large language model pruning via layer collapse.arXiv preprint arXiv:2402.11187, 2024b.
Zellers etal. (2019)Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019.
Zhai etal. (2024)Yuexiang Zhai, Shengbang Tong, Xiao Li, MuCai, Qing Qu, YongJae Lee, and YiMa.Investigating the catastrophic forgetting in multimodal large language model fine-tuning.In Conference on Parsimony and Learning, pp. 202–227. PMLR, 2024.
Zhang etal. (2023a)Biao Zhang, Barry Haddow, and Alexandra Birch.Prompting large language model for machine translation: A case study.In International Conference on Machine Learning, pp. 41092–41110. PMLR, 2023a.
Zhang etal. (2023b)Boyu Zhang, Hongyang Yang, Tianyu Zhou, Muhammad AliBabar, and Xiao-Yang Liu.Enhancing financial sentiment analysis via retrieval augmented large language models.In Proceedings of the fourth ACM international conference on AI in finance, pp. 349–356, 2023b.
Zhao etal. (2024a)Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian.Galore: Memory-efficient llm training by gradient low-rank projection.arXiv preprint arXiv:2403.03507, 2024a.
Zhao etal. (2024b)Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian.Galore: Memory-efficient llm training by gradient low-rank projection, 2024b.URL https://arxiv.org/abs/2403.03507.
Zheng etal. (2024)Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, ZiLin, Zhuohan Li, Dacheng Li, Eric Xing, etal.Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36, 2024.
Zhong etal. (2024)Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzhi Li.Blockpruner: Fine-grained pruning for large language models.arXiv preprint arXiv:2406.10594, 2024.
Zhu etal. (2015)Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.In Proceedings of the IEEE international conference on computer vision, pp. 19–27, 2015.

Appendix A Supplementary Material of Reassessing Layer Pruning in LLMs: New Insights and Methods

Model	Metric	Benchmarks								Avg Acc
Model	Metric	PIQA	HellaSwag	OpenbookQA	ARC-e	ARC-c	MMLU	CMMLU	WinoGrande	Avg Acc
Vicuna-7B-v1.5	Dense	0.7720 $\pm$ 0.0098	0.5642 $\pm$ 0.0049	0.3300 $\pm$ 0.0210	0.7555 $\pm$ 0.0088	0.4326 $\pm$ 0.0145	0.4858 $\pm$ 0.0040	0.3518 $\pm$ 0.0044	0.6953 $\pm$ 0.0129	0.5484
	Reverse-order	0.5642 $\pm$ 0.0116	0.2919 $\pm$ 0.0045	0.1700 $\pm$ 0.0168	0.3258 $\pm$ 0.0096	0.2645 $\pm$ 0.0129	0.4372 $\pm$ 0.0041	0.3069 $\pm$ 0.0043	0.5872 $\pm$ 0.0138	0.3685
	Random	0.5773 $\pm$ 0.0115	0.3083 $\pm$ 0.0046	0.1560 $\pm$ 0.0162	0.3775 $\pm$ 0.0099	0.2176 $\pm$ 0.0121	0.2650 $\pm$ 0.0037	0.2542 $\pm$ 0.0041	0.5067 $\pm$ 0.0141	0.3328
	PPL	0.6572 $\pm$ 0.0111	0.3524 $\pm$ 0.0048	0.1940 $\pm$ 0.0177	0.4971 $\pm$ 0.0103	0.2406 $\pm$ 0.0125	0.2361 $\pm$ 0.0036	0.2510 $\pm$ 0.0040	0.5328 $\pm$ 0.0140	0.3702
	Magnitude-l1	0.5239 $\pm$ 0.0117	0.2585 $\pm$ 0.0044	0.1400 $\pm$ 0.0155	0.2635 $\pm$ 0.0090	0.2184 $\pm$ 0.0121	0.2295 $\pm$ 0.0035	0.2527 $\pm$ 0.0040	0.4893 $\pm$ 0.0140	0.2970
	Magnitude-l2	0.5245 $\pm$ 0.0117	0.2590 $\pm$ 0.0044	0.1300 $\pm$ 0.0151	0.2656 $\pm$ 0.0091	0.2210 $\pm$ 0.0121	0.2293 $\pm$ 0.0035	0.2512 $\pm$ 0.0040	0.4791 $\pm$ 0.0140	0.2950
	BI	0.5250 $\pm$ 0.0117	0.2598 $\pm$ 0.0044	0.1440 $\pm$ 0.0157	0.2740 $\pm$ 0.0092	0.1928 $\pm$ 0.0115	0.2296 $\pm$ 0.0035	0.2476 $\pm$ 0.0040	0.4988 $\pm$ 0.0141	0.2965
	Taylor	0.5283 $\pm$ 0.0116	0.2585 $\pm$ 0.0044	0.1300 $\pm$ 0.0151	0.2572 $\pm$ 0.0090	0.2167 $\pm$ 0.0120	0.2614 $\pm$ 0.0037	0.2513 $\pm$ 0.0040	0.4901 $\pm$ 0.0140	0.2992
Qwen1.5-7B	Dense	0.7845 $\pm$ 0.0096	0.5785 $\pm$ 0.0049	0.3160 $\pm$ 0.0208	0.7125 $\pm$ 0.0093	f0.4053 $\pm$ 0.0143	0.5967 $\pm$ 0.0039	0.7277 $\pm$ 0.0039	0.6575 $\pm$ 0.0133	0.5973
	Reverse-order	0.5783 $\pm$ 0.0115	0.3100 $\pm$ 0.0046	0.1640 $\pm$ 0.0166	0.3047 $\pm$ 0.0094	0.2363 $\pm$ 0.0124	0.2507 $\pm$ 0.0037	0.2564 $\pm$ 0.0041	0.5391 $\pm$ 0.0140	0.3299
	Random	0.6409 $\pm$ 0.0112	0.3268 $\pm$ 0.0047	0.1940 $\pm$ 0.0177	0.4617 $\pm$ 0.0102	0.2261 $\pm$ 0.0122	0.2321 $\pm$ 0.0036	0.2529 $\pm$ 0.0040	0.5083 $\pm$ 0.0141	0.3553
	PPL	0.6529 $\pm$ 0.0111	0.3233 $\pm$ 0.0047	0.1700 $\pm$ 0.0168	0.4360 $\pm$ 0.0102	0.2099 $\pm$ 0.0119	0.2297 $\pm$ 0.0035	0.2541 $\pm$ 0.0041	0.5225 $\pm$ 0.0140	0.3498
	Magnitude-l1	0.5452 $\pm$ 0.0116	0.2690 $\pm$ 0.0044	0.1280 $\pm$ 0.0150	0.2837 $\pm$ 0.0092	0.1962 $\pm$ 0.0116	0.2548 $\pm$ 0.0037	0.2479 $\pm$ 0.0040	0.4862 $\pm$ 0.0140	0.3013
	Magnitude-l2	0.5348 $\pm$ 0.0116	0.2651 $\pm$ 0.0044	0.1520 $\pm$ 0.0161	0.2858 $\pm$ 0.0093	0.1843 $\pm$ 0.0113	0.2659 $\pm$ 0.0037	0.2519 $\pm$ 0.0040	0.5059 $\pm$ 0.0141	0.3057
	BI	0.6001 $\pm$ 0.0114	0.2905 $\pm$ 0.0045	0.1880 $\pm$ 0.0175	0.4099 $\pm$ 0.0101	0.2090 $\pm$ 0.0119	0.2420 $\pm$ 0.0036	0.2472 $\pm$ 0.0040	0.4901 $\pm$ 0.0140	0.3346
	Taylor	0.5223 $\pm$ 0.0117	0.2540 $\pm$ 0.0043	0.1460 $\pm$ 0.0158	0.2403 $\pm$ 0.0088	0.2176 $\pm$ 0.0121	0.2393 $\pm$ 0.0036	0.2478 $\pm$ 0.0040	0.4854 $\pm$ 0.0140	0.2941
Gemma2-2B-It	Dense	0.7867 $\pm$ 0.0096	0.5367 $\pm$ 0.0050	0.3560 $\pm$ 0.0214	0.8085 $\pm$ 0.0081	0.5111 $\pm$ 0.0146	0.5687 $\pm$ 0.0039	0.4499 $\pm$ 0.0045	0.6961 $\pm$ 0.0129	0.5892
	Reverse-order	0.6050 $\pm$ 0.0114	0.3049 $\pm$ 0.0046	0.1900 $\pm$ 0.0176	0.3817 $\pm$ 0.0100	0.2491 $\pm$ 0.0126	0.2327 $\pm$ 0.0036	0.2527 $\pm$ 0.0040	0.5580 $\pm$ 0.0140	0.3468
	Random	0.6741 $\pm$ 0.0109	0.3441 $\pm$ 0.0047	0.2180 $\pm$ 0.0185	0.5446 $\pm$ 0.0102	0.2696 $\pm$ 0.0130	0.2307 $\pm$ 0.0036	0.2540 $\pm$ 0.0041	0.5335 $\pm$ 0.0140	0.3836
	PPL	0.6621 $\pm$ 0.0110	0.3505 $\pm$ 0.0048	0.2380 $\pm$ 0.0191	0.5585 $\pm$ 0.0102	0.2526 $\pm$ 0.0127	0.2328 $\pm$ 0.0036	0.2526 $\pm$ 0.0040	0.5280 $\pm$ 0.0140	0.3844
	Magnitude-l1	0.6649 $\pm$ 0.0110	0.3358 $\pm$ 0.0047	0.1960 $\pm$ 0.0178	0.5564 $\pm$ 0.0102	0.2355 $\pm$ 0.0124	0.2307 $\pm$ 0.0035	0.2516 $\pm$ 0.0040	0.5264 $\pm$ 0.0140	0.3747
	Magnitude-l2	0.6159 $\pm$ 0.0113	0.2956 $\pm$ 0.0046	0.1720 $\pm$ 0.0169	0.4301 $\pm$ 0.0102	0.2073 $\pm$ 0.0118	0.2319 $\pm$ 0.0036	0.2501 $\pm$ 0.0040	0.5178 $\pm$ 0.0140	0.3401
	BI	0.6376 $\pm$ 0.0112	0.3310 $\pm$ 0.0047	0.2140 $\pm$ 0.0184	0.4891 $\pm$ 0.0103	0.2406 $\pm$ 0.0125	0.2397 $\pm$ 0.0036	0.2532 $\pm$ 0.0040	0.5667 $\pm$ 0.0139	0.3715
	Taylor	0.6088 $\pm$ 0.0114	0.3142 $\pm$ 0.0046	0.1880 $\pm$ 0.0175	0.4049 $\pm$ 0.0101	0.2739 $\pm$ 0.0130	0.2297 $\pm$ 0.0035	0.2508 $\pm$ 0.0040	0.5817 $\pm$ 0.0139	0.3565
Llama-3.1-8B-It	Dense	0.8003 $\pm$ 0.0093	0.5910 $\pm$ 0.0049	0.3380 $\pm$ 0.0212	0.8182 $\pm$ 0.0079	0.5179 $\pm$ 0.0146	0.6790 $\pm$ 0.0038	0.5552 $\pm$ 0.0045	0.7395 $\pm$ 0.0123	0.6299
	Reverse-order	0.6376 $\pm$ 0.0112	0.3163 $\pm$ 0.0046	0.1960 $\pm$ 0.0178	0.4019 $\pm$ 0.0101	0.3106 $\pm$ 0.0135	0.2502 $\pm$ 0.0036	0.2482 $\pm$ 0.0040	0.6101 $\pm$ 0.0137	0.3714
	Random	0.5588 $\pm$ 0.0116	0.2730 $\pm$ 0.0044	0.1280 $\pm$ 0.0150	0.2826 $\pm$ 0.0093	0.1903 $\pm$ 0.0115	0.2406 $\pm$ 0.0036	0.2555 $\pm$ 0.0041	0.5020 $\pm$ 0.0141	0.3039
	PPL	0.6643 $\pm$ 0.0110	0.3548 $\pm$ 0.0048	0.1960 $\pm$ 0.0178	0.4718 $\pm$ 0.0102	0.2483 $\pm$ 0.0126	0.2394 $\pm$ 0.0036	0.2446 $\pm$ 0.0040	0.5454 $\pm$ 0.0140	0.3706
	Magnitude-l1	0.5316 $\pm$ 0.0116	0.2576 $\pm$ 0.0044	0.1360 $\pm$ 0.0153	0.2572 $\pm$ 0.0090	0.1980 $\pm$ 0.0116	0.2344 $\pm$ 0.0036	0.2526 $\pm$ 0.0040	0.4933 $\pm$ 0.0141	0.2951
	Magnitude-l2	0.5316 $\pm$ 0.0116	0.2576 $\pm$ 0.0044	0.1360 $\pm$ 0.0153	0.2572 $\pm$ 0.0090	0.1980 $\pm$ 0.0116	0.2344 $\pm$ 0.0036	0.2526 $\pm$ 0.0040	0.4933 $\pm$ 0.0141	0.2951
	BI	0.5773 $\pm$ 0.0115	0.2878 $\pm$ 0.0045	0.1520 $\pm$ 0.0161	0.3674 $\pm$ 0.0099	0.1706 $\pm$ 0.0110	0.2342 $\pm$ 0.0036	0.2466 $\pm$ 0.0040	0.5036 $\pm$ 0.0141	0.3174
	Taylor	0.6088 $\pm$ 0.0114	0.3288 $\pm$ 0.0047	0.1660 $\pm$ 0.0167	0.4318 $\pm$ 0.0102	0.2790 $\pm$ 0.0131	0.2310 $\pm$ 0.0036	0.2534 $\pm$ 0.0041	0.6093 $\pm$ 0.0137	0.3635

Reassessing Layer Pruning in LLMs: New Insights and Methods (5)

Model	Method	Layer	Benchmarks								Avg Acc
Model	Method	Layer	PIQA	HellaSwag	OpenbookQA	ARC-e	ARC-c	MMLU	CMMLU	WinoGrande	Avg Acc
Llama-3.1-8B-It	LoRA	-	0.7138 $\pm$ 0.0105	0.4964 $\pm$ 0.0050	0.2740 $\pm$ 0.0200	0.6848 $\pm$ 0.0095	0.4181 $\pm$ 0.0144	0.2861 $\pm$ 0.0038	0.2504 $\pm$ 0.0040	0.7135 $\pm$ 0.0127	0.4796
	QLoRA	-	0.6496 $\pm$ 0.0111	0.3260 $\pm$ 0.0047	0.1820 $\pm$ 0.0173	0.4520 $\pm$ 0.0102	0.2969 $\pm$ 0.0134	0.3425 $\pm$ 0.0040	0.2627 $\pm$ 0.0041	0.5793 $\pm$ 0.0139	0.3864
	Partial-layer	lm_head only	0.6752 $\pm$ 0.0109	0.3685 $\pm$ 0.0048	0.2100 $\pm$ 0.0182	0.5349 $\pm$ 0.0102	0.3276 $\pm$ 0.0137	0.4315 $\pm$ 0.0041	0.3373 $\pm$ 0.0044	0.6795 $\pm$ 0.0109	0.4456
		lm_head+last layer	0.7029 $\pm$ 0.0107	0.4676 $\pm$ 0.0050	0.2140 $\pm$ 0.0184	0.6393 $\pm$ 0.0099	0.3763 $\pm$ 0.0142	0.5682 $\pm$ 0.0041	0.4483 $\pm$ 0.0046	0.6748 $\pm$ 0.0132	0.5114
		lm_head+last two layers	0.7252 $\pm$ 0.0104	0.5173 $\pm$ 0.0050	0.2800 $\pm$ 0.0201	0.7104 $\pm$ 0.0093	0.4232 $\pm$ 0.0144	0.6058 $\pm$ 0.0040	0.4659 $\pm$ 0.0046	0.7040 $\pm$ 0.0128	0.5540
		lm_head+last three layers	0.7345 $\pm$ 0.0103	0.5290 $\pm$ 0.0050	0.3020 $\pm$ 0.0206	0.7399 $\pm$ 0.0090	0.4360 $\pm$ 0.0145	0.6277 $\pm$ 0.0039	0.4763 $\pm$ 0.0046	0.7151 $\pm$ 0.0127	0.5701

Fine-tuning Method

Model

Method

Iteration steps

Benchmarks

Avg Acc

PIQA

HellaSwag

OpenbookQA

ARC-e

ARC-c

MMLU

CMMLU

WinoGrande

LoRA

Llama-3.1-8B-It

Reverse-order

one-shot

0.6376

\pm

0.0112

0.3163

\pm

0.0046

0.1960

\pm

0.0178

0.4019

\pm

0.0101

0.3106

\pm

0.0135

0.2502

\pm

0.0036

0.2482

\pm

0.0040

0.6101

\pm

0.0137

0.3714

1:8:16

0.6376

\pm

0.0112

0.3160

\pm

0.0046

0.1980

\pm

0.0178

0.3990

\pm

0.0100

0.3106

\pm

0.0135

0.2526

\pm

0.0037

0.2504

\pm

0.0040

0.6046

\pm

0.0137

0.3711

1:1:16

0.6333

\pm

0.0112

0.3259

\pm

0.0047

0.2020

\pm

0.0180

0.4146

\pm

0.0101

0.2961

\pm

0.0133

0.2426

\pm

0.0036

0.2690

\pm

0.0041

0.5912

\pm

0.0138

0.3718

Taylor

one-shot

0.6088

\pm

0.0114

0.3288

\pm

0.0047

0.1660

\pm

0.0167

0.4318

\pm

0.0102

0.2790

\pm

0.0131

0.2310

\pm

0.0036

0.2534

\pm

0.0041

0.6093

\pm

0.0137

0.3635

1:8:16

0.6230

\pm

0.0113

0.3516

\pm

0.0048

0.1480

\pm

0.0159

0.4604

\pm

0.0102

0.2355

\pm

0.0124

0.2541

\pm

0.0037

0.2546

\pm

0.0041

0.5312

\pm

0.0140

0.3573

1:1:16

0.5430

\pm

0.0116

0.2692

\pm

0.0044

0.1580

\pm

0.0163

0.2921

\pm

0.0093

0.1937

\pm

0.0115

0.2334

\pm

0.0036

0.2481

\pm

0.0040

0.5091

\pm

0.0141

0.3058

Gemma2-2B-It

Reverse-order

one-shot

0.6050

\pm

0.0114

0.3049

\pm

0.0046

0.1900

\pm

0.0176

0.3817

\pm

0.0100

0.2491

\pm

0.0126

0.2327

\pm

0.0036

0.2527

\pm

0.0040

0.5580

\pm

0.0140

0.3468

1:6:12

0.6007

\pm

0.0114

0.3076

\pm

0.0046

0.1900

\pm

0.0176

0.3994

\pm

0.0101

0.2483

\pm

0.0126

0.2429

\pm

0.0036

0.2495

\pm

0.0040

0.5478

\pm

0.0140

0.3483

1:1:12

0.6023

\pm

0.0114

0.3173

\pm

0.0046

0.1720

\pm

0.0169

0.3897

\pm

0.0100

0.2449

\pm

0.0126

0.2531

\pm

0.0037

0.2481

\pm

0.0040

0.5387

\pm

0.0140

0.3458

Taylor

one-shot

0.6088

\pm

0.0114

0.3142

\pm

0.0046

0.1880

\pm

0.0175

0.4049

\pm

0.0101

0.2739

\pm

0.0130

0.2297

\pm

0.0035

0.2508

\pm

0.0040

0.5817

\pm

0.0139

0.3565

1:6:12

0.5909

\pm

0.0115

0.2806

\pm

0.0045

0.1380

\pm

0.0154

0.3834

\pm

0.0100

0.2150

\pm

0.0120

0.2295

\pm

0.0035

0.2523

\pm

0.0040

0.5059

\pm

0.0141

0.3245

1:1:12

0.6502

\pm

0.0111

0.3456

\pm

0.0047

0.1860

\pm

0.0174

0.4790

\pm

0.0103

0.2483

\pm

0.0126

0.2314

\pm

0.0036

0.2578

\pm

0.0041

0.5525

\pm

0.0140

0.3689

Partial-layer

Llama-3.1-8B-It

Reverse-order

one-shot

0.6578

\pm

0.0111

0.4137

\pm

0.0049

0.2200

\pm

0.0185

0.5707

\pm

0.0102

0.3294

\pm

0.0137

0.3854

\pm

0.0040

0.3190

\pm

0.0043

0.6504

\pm

0.0134

0.4433

1:1:16

0.6774

\pm

0.0109

0.4164

\pm

0.0049

0.2200

\pm

0.0185

0.5863

\pm

0.0101

0.3362

\pm

0.0138

0.4170

\pm

0.0041

0.3460

\pm

0.0044

0.6385

\pm

0.0135

0.4547

Taylor

one-shot

0.6649

\pm

0.0110

0.3985

\pm

0.0049

0.2100

\pm

0.0182

0.5581

\pm

0.0102

0.3251

\pm

0.0137

0.3054

\pm

0.0039

0.2876

\pm

0.0042

0.6212

\pm

0.0136

0.4214

1:1:16

0.5876

\pm

0.0115

0.2813

\pm

0.0045

0.1300

\pm

0.0151

0.3986

\pm

0.0100

0.1980

\pm

0.0116

0.2508

\pm

0.0037

0.2502

\pm

0.0040

0.4957

\pm

0.0141

0.3240

Reassessing Layer Pruning in LLMs: New Insights and Methods (6)

Reassessing Layer Pruning in LLMs: New Insights and Methods (7)

Model	Metric	Calibration Samples	Removed Layers	Benchmarks								Avg Acc
Model	Metric	Calibration Samples	Removed Layers	PIQA	HellaSwag	OpenbookQA	ARC-e	ARC-c	MMLU	CMMLU	WinoGrande	Avg Acc
Llama-3.1-8B-Instruct	BI	1	2,3,5,6,7,8,11,12	0.7029 $\pm$ 0.0107	0.4167 $\pm$ 0.0049	0.2060 $\pm$ 0.0181	0.6136 $\pm$ 0.0100	0.2739 $\pm$ 0.0130	0.2362 $\pm$ 0.0036	0.2512 $\pm$ 0.0040	0.5225 $\pm$ 0.0140	0.40
		5	3,4,5,8,9,10,13,19	0.7236 $\pm$ 0.0104	0.4400 $\pm$ 0.0050	0.2420 $\pm$ 0.0192	0.6730 $\pm$ 0.0096	0.3311 $\pm$ 0.0138	0.2524 $\pm$ 0.0037	0.2553 $\pm$ 0.0041	0.5485 $\pm$ 0.0140	0.43
		10	2,3,4,5,6,7,8,9	0.7176 $\pm$ 0.0105	0.4196 $\pm$ 0.0049	0.2020 $\pm$ 0.0180	0.6107 $\pm$ 0.0100	0.2841 $\pm$ 0.0132	0.2417 $\pm$ 0.0036	0.2494 $\pm$ 0.0040	0.5391 $\pm$ 0.0140	0.41
		30	2,3,4,10,11,12,13,14	0.7209 $\pm$ 0.0105	0.4328 $\pm$ 0.0049	0.2040 $\pm$ 0.0180	0.6414 $\pm$ 0.0098	0.3259 $\pm$ 0.0137	0.2500 $\pm$ 0.0036	0.2576 $\pm$ 0.0041	0.5517 $\pm$ 0.0140	0.42
		50	2,3,4,5,6,7,10,13	0.7100 $\pm$ 0.0106	0.4091 $\pm$ 0.0049	0.2180 $\pm$ 0.0185	0.6221 $\pm$ 0.0099	0.2875 $\pm$ 0.0132	0.2492 $\pm$ 0.0036	0.2529 $\pm$ 0.0040	0.5462 $\pm$ 0.0140	0.41
	Taylor	1	27, 26, 25, 24, 28, 23, 29, 22	0.6088 $\pm$ 0.0114	0.3288 $\pm$ 0.0047	0.1660 $\pm$ 0.0167	0.4318 $\pm$ 0.0102	0.2790 $\pm$ 0.0131	0.2310 $\pm$ 0.0036	0.2534 $\pm$ 0.0041	0.6093 $\pm$ 0.0137	0.36
		5	24, 26, 25, 28, 27, 23, 29, 22	0.6088 $\pm$ 0.0114	0.3288 $\pm$ 0.0047	0.1660 $\pm$ 0.0167	0.4318 $\pm$ 0.0102	0.2790 $\pm$ 0.0131	0.2310 $\pm$ 0.0036	0.2534 $\pm$ 0.0041	0.6093 $\pm$ 0.0137	0.36
		10	24, 26, 25, 28, 27, 23, 29, 22	0.6088 $\pm$ 0.0114	0.3288 $\pm$ 0.0047	0.1660 $\pm$ 0.0167	0.4318 $\pm$ 0.0102	0.2790 $\pm$ 0.0131	0.2310 $\pm$ 0.0036	0.2534 $\pm$ 0.0041	0.6093 $\pm$ 0.0137	0.36
		30	24, 23, 25, 26, 22, 27, 28, 20	0.7280 $\pm$ 0.0104	0.4985 $\pm$ 0.0050	0.2460 $\pm$ 0.0193	0.6961 $\pm$ 0.0094	0.4130 $\pm$ 0.0144	0.6611 $\pm$ 0.0038	0.4915 $\pm$ 0.0046	0.7032 $\pm$ 0.0128	0.55
		50	24, 23, 25, 26, 22, 27, 28, 20	0.7280 $\pm$ 0.0104	0.4985 $\pm$ 0.0050	0.2460 $\pm$ 0.0193	0.6961 $\pm$ 0.0094	0.4130 $\pm$ 0.0144	0.6611 $\pm$ 0.0038	0.4915 $\pm$ 0.0046	0.7032 $\pm$ 0.0128	0.55

Model	Generated Sentences
Llama-3.1-6.3B-It-Alpaca	To be or not to be, that is the question that has been asked thousands of times throughout history. but have you ever wondered what it would be like to not exist? what would it feel like to not have consciousness or awareness of any kind? would it be like being asleep or a combed out hair? no, it would be something else entirely. it would be like not being here at all, like being a tree or a rock, a piece of furniture or a piece of trash. nothingness is not something we can really comprehend because we can’t experience it. we can’t even imagine it because we can’t imagine.
	It fills me with such pride and joy knowing that I have been able to help shaped the lives of countless students over the years. I am proud of the numerous accomplishments and achievements of my students, and I am proud of the role I have played in shaping them into the people they are today. As a teacher, I believe it is my duty to provide an optimal learning environment that fosters growth and development of my students. I believe in making learning fun and engaging, and I am dedicated to pushing the bounds of innovation in the classroom. I am dedicated to ensuring that my students leave my classroom with the knowledge and skills they need …
	What’s great about the holiday season? Christmas lights, Christmas shows, Christmas presents, Christmas holiday traditions. But what’s not great about the holiday season? crowds, stress, Santa Claus, Christmas holiday stress, Christmas holiday stressors. It’s important to remember to do things that help you relax during the holiday season, such as taking time for yourself, engaging in relaxation techniques, practicing mindfulness, engaging in physical activity, practicing gratitude, practicing self-care, engaging in activities that bring you joy, and spending time with loved ones. These are all important components of stressors prevention during the holiday season. Here are some tips to help you. …
Llama-3.1-6.3B-It-Dolly	To be or not to be, that is the question - Hamlet’s famous soliloquy - could easily apply to my current state of mind. I’ve been struggling to decision whether to go back to school or not. On one hand, I’m really interesting in learning more about the subject matter I’ve been passionate about for years, and I think pursuing a degree would be a great way to formalize my knowledge. On the Hamlet scale, this option would be ”to be”: me myself, fully engaged in learning and growing as a person. On the other hand, I’ve been married for almost 20 years …
	It fills me with such pride and joy to see my little human growing up. yesterday was a special day in our household because we celebrated one of the major milestone in life. my little one cut his first teeth. Yes you heard it right he is now a big boy and his first teeth are now visible. It has been a long journey since we started noticed his teething symptoms and we have been through lot of sleepless nights and teething tantrums but it all pay off now. I can see the relief in his little face when he finally gets to experience the world without the constant pain in his little bottom teeth. As a …
	What’s great about the holiday season is spending time with loved ones and making new traditions. But let’s not forget about the stress and pressure that comes with it. We’re here to help you beat stress during the holiday season with some tips from our #Ask a therapist series with licensed therapist, Dr. Dezympfer. ### tip #1: Prioritize Self-Cark. With all the commotion during the holiday season, it can be easy to forget to take care of yourself. Make sure to schedule time for self-Cark. whether that is taking a yoga class, taking a few minutes each day to meditate, …
Llama-3.1-8B-It	To be or not to be, that is the question. How does one decide whether a being is alive or not? The issue of what constitutes life has puzzled philosophers and scientists alike for centuries. A being is alive if it meets the criteria of life, which are: Metabolism: The ability to carry out basic functions such as respiration, photosynthesis, and nutrient uptake.Homeostasis: The ability to maintain a stable internal environment, despite changes in the external environment.Growth and Development: The ability to increase in size and complexity over time.Reproduction: The ability to produce offspring, either sexually or asex
	It fills me with such pride and joy to see my friends’ passions ignite into flame and to watch them soar to great heights! It was an absolutely honour to have been able to share this experience with you and to bear witness to your hard work and dedication! It was an absolute pleasure to get to know you and share our experiences, but I know that this is just the beginning of an amazing journey! I feel so grateful to have you in my life, and I am constantly in awe of your love, kindness, and generosity! It’s been such an incredible journey watching you discover and grow, and I feel so lucky to have…
	What’s great about the holiday season? In addition to being able to spend time with loved ones and get some much-needed R&R, many people enjoy the idea of giving back to others. Whether it’s volunteering, donating to charity, or participating in a Secret Santa gift exchange, the holiday season can be a time of kindness and generosity. But have you ever thought about how you might be able to combine your love of cooking and giving back this holiday season? If so, you might be interested in hosting a charity-themed potluck dinner or bake sale. Here are a few ideas to get you started: Host a potluck dinner to…