Evaluating LLMs with N-Day-Bench for Real-World Vulnerability Discovery

N-Day-Bench is a cutting-edge benchmark designed to assess the ability of advanced language models to uncover real-world vulnerabilities, known as 'N-Days', that have been disclosed after the models' knowledge cut-off dates. By providing a uniform testing environment, this benchmark ensures that all models are evaluated fairly, without the possibility of manipulating results through reward hacking. The primary focus is on assessing the cybersecurity capabilities of large language models, particularly their proficiency in vulnerability discovery.

The benchmark is dynamic, with test cases being refreshed monthly to incorporate the latest vulnerabilities, and models are updated to their newest versions and checkpoints. This ensures that the evaluation remains relevant and challenging, reflecting the ever-evolving landscape of cybersecurity threats. Additionally, all test traces are made publicly accessible, promoting transparency and collaboration within the research community. This initiative is spearheaded by Winfunc Research, highlighting their commitment to advancing cybersecurity through innovative AI applications.