Enhancing Code Generation Models with Simple Self-Distillation

In the realm of advanced coding tasks, the scarcity of high-quality supervised data is a significant hurdle. Traditional methods like teacher-based distillation and reinforcement learning face limitations, prompting the exploration of unsupervised alternatives. Enter Simple Self-Distillation (SSD), a method that allows models to enhance their performance using only their raw outputs. By sampling solutions from the base model with specific temperature settings and fine-tuning on these unverified samples, SSD eliminates the need for external labeled data or complex verification processes.

Remarkably, SSD has demonstrated significant improvements in code generation tasks. For instance, the Qwen3-30B-Instruct model's pass@1 score on LiveCodeBench v6 increased from 42.4% to 55.3%, with even greater gains on challenging problems. This method is not limited to a single model; it generalizes across multiple models and scales, highlighting its versatility.

The effectiveness of SSD can be attributed to its ability to navigate the precision-exploration conflict inherent in code generation. Code tasks involve 'fork' positions, where multiple solutions are plausible, and 'lock' positions, where syntax and semantics are more rigid. SSD reshapes the model's distributions, suppressing distractors at locks while maintaining diversity at forks. This nuanced approach allows for better exploration without compromising accuracy.

Our findings suggest that existing code models possess untapped potential that can be unlocked through SSD, bypassing the need for traditional reinforcement learning or teacher models. This method not only enhances performance but also reveals latent capabilities within code generation models.