---

Collective Knowledge (CK) is a community initiative focused on understanding how to run AI, ML, and other emerging workloads efficiently and cost-effectively in real-world scenarios, across diverse models, datasets, software, and hardware, leveraging state-of-the-art R&D and public challenges.

This educational project was created by Grigori Fursin, cTuning and cTuning Labs in collaboration with ACM, IEEE, HiPEAC, MLCommons, dedicated volunteers and contributors, and participants in open challenges.

You can learn more about this open-source project through the ACM TechTalk'21, the Journal of Royal Society'20 article, the ACM REP'23 keynote, and our white paper'24.

---

Explore our past prototypes of AI/ML benchmarking analytics (2024-2025):

• See our open MLPerf dataset at HuggingFace with white paper
• See our performance & cost analysis for AI training and inference workloads based on our open MLPerf dataset
• See our research dashboard to analyze optimal SW/HW configurations for AI workloads based on our open MLPerf dataset

Explore our legacy prototypes, developed in collaboration with cTuning, MLCommons, and ACM (2022-2025):

• Install our legacy CLI-based virtual MLOps automation framework (CM/MLC/MLCflow/MLPerf/CMX)
• Browse MLPerf and other results
• Run MLPerf via CMX (MLCommons website)
• Reuse virtual MLOps and MLPerf automations (legacy CM & MLC)
• See low-level CM/CMX Python API

Explore our community initiatives on collaborative optimization and reproducibility of AI systems:

• See our public hackathons, challenges and reproducibility events
• See our contributors and the leaderboard

---

Explore our past open-source technologies developed by Grigori Fursin in collaboration with our valued partners and the community:

• 2021-2025: CM automation language and framework (cTuning and MLCommons)
• 2025: Prototype of FlexBench to benchmark vLLM across diverse hardware, software and HuggingFace models using the MLPerf methodology and CM automation: MLPerf inference v5.0 GitHub, Open MLPerf dataset at HuggingFace
• 2024-2025: CMX4MLOps automations (MLCommons, FlexAI and cTuning)
• 2022-2024: CM4MLOps automations (MLCommons, OctoAI and cTuning)
• 2021: CM-MLOps automations (MLCommons, OctoAI and cTuning)
• 2019-2021: Collective Knowledge portable AI/ML solutions
• 2017-2020: Android app to crowd-benchmark deep learning models
• 2015-2021: Collective Knowledge automation framework v1
• 2015-2021: Collective Knowledge crowd-optimization results
• 2011-2014: Prototype of a public optimization repository, portable benchmarking pipeline example
• 2007-2011: Collective Tuning portal for self-optimizing compilers including the MILEPOST GCC