Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V3.5.1 #1372

Merged
merged 3 commits into from
Dec 3, 2024
Merged

V3.5.1 #1372

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3aa5a15
## V3.5.1
gfursin Nov 29, 2024
81fdf86
Merge pull request #40 from mlcommons/master
gfursin Nov 29, 2024
dfc74a6
## V3.5.1
gfursin Dec 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 95 additions & 43 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,76 +9,126 @@
[![MLPerf inference resnet50](https://github.com/mlcommons/ck/actions/workflows/test-cm-mlperf-inference-resnet50.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cm-mlperf-inference-resnet50.yml)
[![CMX: image classification with ONNX](https://github.com/mlcommons/ck/actions/workflows/test-cmx-image-classification-onnx.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cmx-image-classification-onnx.yml)

### About
## Collective Knowledge

[Collective Knowledge (CK, CM, CM4MLOps, CM4MLPerf and CMX)](https://cKnowledge.org)
is an educational community project to learn how to run AI, ML and other emerging workloads
in the most efficient and cost-effective way across diverse models, data sets, software and hardware.
It includes the following sub-projects.

CK consists of several sub-projects:
### Collective Mind (CM)

* [Collective Mind framework (CM)](cm) - a very lightweight Python-based framework with minimal dependencies
intended to help researchers and engineers automate their repetitive, tedious and time-consuming tasks
to build, run, benchmark and optimize AI, ML and other applications and systems
across diverse and continuously changing models, data, software and hardware.
[Collective Mind (CM)](https://github.com/mlcommons/ck/tree/master/cm) -
a very lightweight Python-based framework with a unified CLI, Python API and minimal dependencies
intended to help researchers and engineers automate their repetitive, tedious and time-consuming tasks
to build, run, benchmark and optimize AI, ML and other applications and systems
across diverse and continuously changing models, data, software and hardware.

* [CM4MLOPS](https://github.com/mlcommons/cm4mlops) -
a collection of portable, extensible and technology-agnostic automation recipes
with a common CLI and Python API (CM scripts) to unify and automate
all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications
on diverse platforms with any software and hardware: see [online catalog at CK playground](https://access.cknowledge.org/playground/?action=scripts),
[online MLCommons catalog](https://docs.mlcommons.org/cm4mlops/scripts)
Collective Mind is continuously enhanced through public and private CM4* Git repositories,
which serve as the unified interface for various collections of reusable automations and artifacts.

* [CM interface to run MLPerf inference benchmarks](https://docs.mlcommons.org/inference)
The CM architecture diagram is available for viewing
[here](https://github.com/mlcommons/ck/tree/master/docs/specs/cm-diagram-v3.5.1.png).

* [CM4ABTF](https://github.com/mlcommons/cm4abtf) - a unified CM interface and automation recipes
to run automotive benchmark across different models, data sets, software and hardware from different vendors.
### Collective Mind repositories

* [CMX (the next generation of CM, CM4MLOps and CM4MLPerf)](cm/docs/cmx) -
we are developing the next generation of CM
to make it simpler and more flexible based on user feedback. Please follow
this project [here]( https://github.com/orgs/mlcommons/projects/46 ).
#### CM4MLOps

[CM4MLOPS repository powered by CM](https://github.com/mlcommons/cm4mlops) -
a collection of portable, extensible and technology-agnostic automation recipes
with a common CLI and Python API (CM scripts) to unify and automate
all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications
on diverse platforms with any software and hardware.

* [Collective Knowledge Playground](https://access.cKnowledge.org) - a unified platform
to list CM scripts similar to PYPI, aggregate AI/ML Systems benchmarking results in a reproducible format with CM workflows,
and organize [public optimization challenges and reproducibility initiatives](https://access.cknowledge.org/playground/?action=challenges)
to co-design more efficient and cost-effiective software and hardware for emerging workloads.
The two key automations are *script" and *cache*:
see [online catalog at CK playground](https://access.cknowledge.org/playground/?action=scripts),
[online MLCommons catalog](https://docs.mlcommons.org/cm4mlops/scripts).

* [CM4MLPerf-results](https://github.com/mlcommons/cm4mlperf-results) -
a simplified and unified representation of the past MLPerf results
for further visualization and analysis using [CK graphs](https://access.cknowledge.org/playground/?action=experiments)
(*the new version is coming soon*).
CM scripts extend the concept of `cmake` with simple Python automations, native scripts
and JSON/YAML meta descriptions. They require Python 3.7+ with minimal dependencies and are
[continuously extended by the community and MLCommons members](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)
to run natively on Ubuntu, MacOS, Windows, RHEL, Debian, Amazon Linux
and any other operating system, in a cloud or inside automatically generated containers
while keeping backward compatibility.

See the [online documentation](https://docs.mlcommons.org/inference)
at MLCommons to run MLPerf inference benchmarks across diverse systems using CM.

* [Artifact Evaluation](https://cTuning.org/ae) - automating artifact evaluation and reproducibility initiatives at ML and systems conferences.
#### CM4ABTF

### Deprecated and archived projects
[CM4ABTF repository powered by CM](https://github.com/mlcommons/cm4abtf) -
a collection of portable automations and CM scripts to run the upcoming
automotive MLPerf benchmark across different models, data sets, software
and hardware from different vendors.

* [CM-MLOps](cm-mlops) - now [CM4MLOps](cm4mlops)
* [CK automation framework v1 and v2](ck) - now [CM](cm)
#### CM4MLPerf-results

[CM4MLPerf-results powered by CM](https://github.com/mlcommons/cm4mlperf-results) -
a simplified and unified representation of the past MLPerf results
in the CM format for further visualization and analysis using [CK graphs](https://access.cknowledge.org/playground/?action=experiments).

### License
#### CM4Research

[CM4Research repository powered by CM](https://github.com/ctuning/cm4research) -
a unified interface designed to streamline the preparation, execution, and reproduction of experiments in research projects.


### Projects powered by Collective Mind

#### Collective Knowledge Playground

[Collective Knowledge Playground](https://access.cKnowledge.org) -
a unified and open-source platform designed to [index all CM scripts](https://access.cknowledge.org/playground/?action=scripts) similar to PYPI,
assist users in preparing CM commands to:

* [run MLPerf benchmarks](https://access.cknowledge.org/playground/?action=howtorun)
* aggregate, process, visualize, and compare [benchmarking results](https://access.cknowledge.org/playground/?action=experiments) for AI and ML systems
* organize [open, reproducible optimization challenges and tournaments](https://access.cknowledge.org/playground/?action=challenges).

These initiatives aim to help academia and industry
collaboratively enhance the efficiency and cost-effectiveness of AI systems.

#### Artifact Evaluation

[Artifact Evaluation automation](https://cTuning.org/ae) - a community-driven initiative
leveraging the Collective Mind framework to automate artifact evaluation
and support reproducibility efforts at ML and systems conferences.


## Incubator

[CMX](https://github.com/mlcommons/ck/tree/master/cmx) - the next evolution of the Collective Mind framework,
designed to enhance simplicity, flexibility, and extensibility of automations
based on user feedback. Follow the project's progress [here]( https://github.com/orgs/mlcommons/projects/46 ).


## Archived projects

* [CM-MLOps](https://github.com/mlcommons/ck/tree/master/cm-mlops) - now [CM4MLOps](https://github.com/mlcommons/ck/tree/master/cm4mlops)
* [CK automation framework v1 and v2](https://github.com/mlcommons/ck/tree/master/ck) - now [CM](https://github.com/mlcommons/ck/tree/master/cm)


## License

[Apache 2.0](LICENSE.md)

### Copyright
## Copyright

* Copyright (c) 2021-2024 MLCommons
* Copyright (c) 2014-2021 cTuning foundation

### Author
## Author

* [Grigori Fursin](https://cKnowledge.org/gfursin) (FlexAI, cTuning)

### Maintainers
## Maintainers

* Collective Mind (CM): [Grigori Fursin](https://cKnowledge.org/gfursin)
* CM4MLOps repository: [Arjun Suresh](https://github.com/arjunsuresh) and [Anandhu Sooraj](https://github.com/anandhu-eng)
* CMX (the next generation of CM and CM4MLOps): [Grigori Fursin](https://cKnowledge.org/gfursin)

* [Collective Mind (CM)](cm): [Grigori Fursin](https://cKnowledge.org/gfursin)
* CM4MLOps (CM automation recipes): [Arjun Suresh](https://github.com/arjunsuresh) and [Anandhu Sooraj](https://github.com/anandhu-eng)
* CMX (the next generation of CM, CM4MLOps and CM4MLPerf): [Grigori Fursin](https://cKnowledge.org/gfursin)

### Citing our project
## Citing Collective Mind and Collective Knowledge

If you found the CM automation framework helpful, kindly reference this article:
[ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ].
Expand All @@ -91,17 +141,19 @@ To learn more about the motivation behind CK and CM technology, please explore t
* Journal of Royal Society'20: [ [paper](https://royalsocietypublishing.org/doi/10.1098/rsta.2020.0211) ]


### CM Documentation
## CM Documentation

* [CM installation GUI](https://access.cknowledge.org/playground/?action=install)
* [Collective Mind white paper](https://arxiv.org/abs/2406.16791)
* [CM/CMX architecture](https://github.com/mlcommons/ck/tree/master/docs/specs/cm-diagram-v3.5.1.png)
* [CM/CMX installation GUI](https://access.cknowledge.org/playground/?action=install)
* [CM Getting Started Guide and FAQ](https://github.com/mlcommons/ck/tree/master/docs/getting-started.md)
* [Common CM interface to run MLPerf inference benchmarks](https://github.com/mlcommons/ck/tree/master/docs/mlperf/inference)
* [Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23](https://github.com/mlcommons/ck/tree/master/docs/tutorials/common-interface-to-reproduce-research-projects.md)
* [CM automation recipes for MLOps and DevOps](https://access.cknowledge.org/playground/?action=scripts)
* [CM4MLOps automation recipes for MLOps and DevOps](https://access.cknowledge.org/playground/?action=scripts)
* [Other CM tutorials](https://github.com/mlcommons/ck/tree/master/docs/tutorials)
* [Full documentation](https://github.com/mlcommons/ck/tree/master/docs/README.md)
* [CM taskforce](https://github.com/mlcommons/ck/tree/master/docs/taskforce.md)
* [CMX, CM and CK history](https://github.com/mlcommons/ck/tree/master/docs/history.md)
* [CK, CM and CMX history](https://github.com/mlcommons/ck/tree/master/docs/history.md)


### Acknowledgments
Expand Down
10 changes: 8 additions & 2 deletions cm/CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
## V3.4.4.1
- added tabulate as requirement
## V3.5.1
- added Collective Mind architecture diagram:
https://github.com/mlcommons/ck/tree/master/docs/specs/cm-diagram-v3.5.1.png
- added tabulate as requirement based on user feedback
- updated white paper: https://arxiv.org/abs/2406.16791
- merged Anandu's docs update
- added links to archived repositories to keep development history
(ck, cm-mlops, etc)

## V3.4.4
- improved error reporting in utils.load_json and utils.load_yaml
Expand Down
Loading
Loading