Open date: 2023 Jul 4
Closing date: 2023 Aug 17
Collective Knowledge Contributor award: Yes
Our goal is to help the community benchmark and optimize various AI/ML applications across diverse software and hardware provided by volunteers similar to SETI@home!
Open-source MLPerf inference benchmarks were developed by a consortium of 50+ companies and universities (MLCommons) to enable trustable and reproducible comparison of AI/ML systems in terms of latency, throughput, power consumption, accuracy and other metrics across diverse software/hardware stacks from different vendors.
However, running MLPerf inference benchmarks and submitting results turned out to be a challenge even for experts and could easily take many weeks to prepare. That's why MLCommons, cTuning.org and cKnowledge.org decided to develop an open-source, technology-agnostic and non-intrusive Collective Mind automation language (CM) and Collective Knowledge Playground (CK) to help anyone run, reproduce, optimize and compare MLPerf inference benchmarks out-of-the-box across diverse software, hardware, models and data sets.
You can read more about our vision, open-source technology and future plans in this presentation.
We would like to ask volunteers run various MLPerf inference benchmarks on diverse CPUs (Intel, AMD, Arm) and Nvidia GPUs similar to SETI@home across different framework (ONNX, PyTorch, TF, TFLite) either natively or in a cloud (AWS, Azure, GCP, Alibaba, Oracle, OVHcloud, ...) and submit results to MLPerf inference v3.1.
However, since some benchmarks may take 1..2 days to run, we suggest to start in the following order (these links describe CM commands to run benchmarks and submit results):
CPU: Reference implementation of Image Classification with ResNet50 (open and then closed division)
Nvidia GPU: Nvidia optimized implementation of all other models (open and closed division)
Please read this documentation to set up and run above benchmarks using CM.
You can register your participation for the Collective Knowledge leaderboard using this guide.
Please report encountered problems using GitHub issues to help the community improve the portability of the CM automation for MLPerf and other benchmarks and projects.
Looking forward to your submissions and happy hacking!
You can see shared results in this repostiory with PRs from participants here.
All accepted results will be publicly available in the CM format with derived metrics in this MLCommons repository, in MLCommons Collective Knowledge explorer and at official MLCommons website.