COFFEQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answering
arXiv:2610.12837v1 Announce Type: cross Abstract: Knowledge distillation aims to train a smaller student model to mimic a larger teacher model for resource-constrained environments. However, the process of distillation is typically data-intensive and computationally expensive, particularly for generative tasks. Few-shot distillation offers an alternative by transferring teacher knowledge using limited training samples. However, we observe that few-shot distillation performance is heavily depend…