Open date: 2023 Feb 1
Closing date: 2023 Mar 1
Shared experiments:
Run MLPerf inference v3.0 benchmarks out-of-the-box across diverse implementations, software and hardware using the MLCommons CM automation language and submit public results to the MLPerf inference v3.0 via cTuning foundation.
This challenge has been successfully completed.
Official results: * https://github.com/mlcommons/inference_results_v3.0/tree/main/closed/cTuning * https://github.com/mlcommons/inference_results_v3.0/tree/main/open/cTuning
Results in the MLCommons CK/CM format: * https://github.com/mlcommons/cm4mlperf-results
Visualization and comparison with derived metrics: * MLCommons Collective Knowledge Playground.
We are very pleased to announce the successful outcome of the 1st community challenge to run, reproduce and optimize MLPerf inference v3.0 benchmarks: our MLCommons CK/CM workflow automation framework has helped to prepare more than 80% of all submission results including 98% of power results with very diverse technology and benchmark implementations from Neural Magic, Qualcomm, cKnowledge Ltd, KRAI, cTuning foundation, Dell Technologies, Hewlett Packard Enterprise, Lenovo, Hugging Face, NVIDIA, Intel Corporation, AMD and Apple across diverse CPUs, GPUs and DSPs with PyTorch, ONNX, QAIC, TF/TFLite, TVM and TensorRT using popular cloud providers (GCP, AWS, Azure) and individual servers and edge devices provided by our volunteers.
You can now see and compare all MLPerf inference results v3.0, v2.1 and v2.0 online together with reproducibility reports including the MLPerf BERT model from the Hugging Face Zoo on Nvidia Jetson Orin platform. You can even create your own derived metrics (such as performance per Watt), provide your own constraints using this MLCommons repository and visualize them as shown in this example.
Additional thanks to Michael Goin from Neural Magic, our international students including Himanshu Dutta, Aditya Kumar Shaw, Sachin Mudaliyar, Thomas Zhu, and all CK/CM users and contributors for helping us to validate, use and improve this open-source technology to automate benchmarking and optimization of AI/ML systems in terms of performance, accuracy, power and costs! We are also grateful to HiPEAC and OctoML for sponsoring initial development and Peter Mattson, David Kanter, Vijay Janapa Reddi and Alexandros Karargyris for fruitful discussions.