Rebuilding Trust in Open-Source Models with Kimi Vendor Verifier

With the release of the Kimi K2.6 model, we are excited to introduce the Kimi Vendor Verifier (KVV) project, an open-source initiative aimed at ensuring the accuracy of inference implementations for open-source models. This project emerged from our realization that simply open-sourcing a model isn't enough; it must also function correctly across various platforms. We discovered systemic issues in benchmark scores, often due to misuse of decoding parameters, prompting us to enforce specific constraints at the API level.

Through extensive testing, we identified a widespread problem: as model weights become more open and deployment channels diversify, maintaining quality becomes challenging. To address this, we developed six critical benchmarks designed to expose infrastructure failures. These include pre-verification of API parameters, a quick OCRBench for multimodal pipelines, and MMMU Pro for testing diverse visual inputs. AIME2025 serves as a long-output stress test, while K2VV ToolCall and SWE-Bench focus on trigger consistency and coding tests, respectively.

Our approach includes embedding with vLLM/SGLang/KTransformers communities to address root causes and offering pre-release validation to allow infrastructure providers to test models before deployment. Continuous benchmarking with a public leaderboard will promote transparency and accuracy among vendors.

Testing the full evaluation workflow on NVIDIA H20 8-GPU servers took about 15 hours, but we've optimized scripts for efficiency in long-running scenarios. We invite more vendors to join us in expanding coverage and developing lighter tests. By sharing both the weights and the knowledge to run them correctly, we aim to fortify trust in the open-source ecosystem.