Skip to content

Commit

Permalink
clean up (#1351)
Browse files Browse the repository at this point in the history
  • Loading branch information
ctuning-admin authored Nov 11, 2024
2 parents 45972c7 + b1a3d57 commit 342cc44
Show file tree
Hide file tree
Showing 5 changed files with 160 additions and 11 deletions.
25 changes: 16 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,13 +62,17 @@ CK consists of several sub-projects:
* CM/CMX/CM4Research: [Grigori Fursin](https://cKnowledge.org/gfursin)
* CM4MLOps: [Arjun Suresh](https://github.com/arjunsuresh) and [Anandhu Sooraj](https://github.com/anandhu-eng)

### Motivation and long-term vision
### Citing our project

You can learn more about the motivation behind these projects from the following articles and presentations:
If you found the CM automation framework helpful, kindly reference this article:
[ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ].

* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ]
To learn more about the motivation behind CK and CM technology, please explore the following presentations:

* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ]
* ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ]
* ACM TechTalk'21 about automating research projects: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ]
* ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ]


### CM Documentation

Expand All @@ -85,8 +89,11 @@ You can learn more about the motivation behind these projects from the following

### Acknowledgments

The Collective Mind automation framework (CM) was created by [Grigori Fursin](https://cKnowledge.org/gfursin),
sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons to benefit everyone.
This open-source technology (CM, CM4MLOps, CM4MLPerf, CM4ABTF, CM4Research, etc)
is being developed as a community effort thanks to all our fantastic
[volunteers, collaborators and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)!
The open-source Collective Knowledge project (CK)
was founded by [Grigori Fursin](https://cKnowledge.org/gfursin),
sponsored by cTuning.org, HiPEAC and OctoML, and donated to MLCommons
to serve the wider community. This open-source initiative includes
CM, CM4MLOps/CM4MLPerf, CM4ABTF, and CMX automation technologies.
Its development is a collaborative community effort,
made possible by our dedicated [volunteers, collaborators,
and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)!
2 changes: 1 addition & 1 deletion cm4abtf/README.md
Original file line number Diff line number Diff line change
@@ -1 +1 @@
TBD
Development CM repository: https://github.com/mlcommons/cm4abtf
2 changes: 1 addition & 1 deletion cm4mlops/README.md
Original file line number Diff line number Diff line change
@@ -1 +1 @@
TBD
Development repository: https://github.com/mlcommons/cm4mlops
16 changes: 16 additions & 0 deletions cmx/README.md
Original file line number Diff line number Diff line change
@@ -1 +1,17 @@
# Collective Mind v3 aka CMX

CMX represents the next generation of the CM automation framework.
Developed by Grigori Fursin, it builds upon valuable feedback
gathered from CM users.


## Documentation

* [Installation (Linux, Windows, MacOS)](install.md)

TBD


## Contacts

* Contact the CM/CMX author and maintainer for more details: [Grigori Fursin](https://cKnowledge.org/gfursin)
126 changes: 126 additions & 0 deletions cmx/install.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
[ [Back to index](README.md) ]

# CM Installation

CM framework requires minimal dependencies to run on any platform: `python 3.7+, pip, venv, git, git-lfs, wget, curl`.

By default, CM will pull Git repositories and cache installations and downloaded files in your `$HOME/CM` directory on Linux and MacOS
or `%userprofile%\CM` directory on Windows.
You can change it to any another directory using the `CM_REPOS` environment variable, for example `export CM_REPOS=/scratch/CM`.

*Feel free to use the [online installation GUI](https://access.cknowledge.org/playground/?action=install-cmx)*.


## Ubuntu, Debian

*We have successfully tested CMX with the following system dependencies on Ubuntu 20.x, 22.x , 24.x:*

```bash
sudo apt update
sudo apt install python3 python3-pip python3-venv git git-lfs wget curl

python3 -m venv cm
source cm/bin/activate

pip install cmind
```

Note that you may need to restart your shell to update PATH to the "cmx" binary.

Alternatively you can run

```bash
source $HOME/.profile
```

You can now check that all system dependencies are installed using the following command:
```bash
cmx init
```
You can also check that CMX core works using the following command:
```bash
cmx test core
```

## Red Hat

*We have successfully tested CM on Red Hat 9 and CentOS 8*

```bash
sudo dnf update

sudo dnf install python3 python-pip git git-lfs wget curl

python3 -m pip install cmind --user

```


## MacOS

*Note that CM currently does not work with Python installed from the Apple Store.
Please install Python via brew as described below.*

If `brew` package manager is not installed, please install it as follows (see details [here](https://brew.sh/)):
```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```

Don't forget to add brew to PATH environment as described in the end.

Then install python, pip, git and wget:

```bash
brew install python3 git git-lfs wget curl

python3 -m pip install cmind
```

*Sometimes python does not add `cm` and `cmr` binaries to the `PATH` environment variable.
You may need to find these files and add their path to `PATH` variable.
We plan to simplify this installation in the future.*


## Windows

* Configure Windows 10+ to [support long paths](https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#enable-long-paths-in-windows-10-version-1607-and-later) from command line as admin:
<small>
<small>
<small>
```bash
reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem" /v LongPathsEnabled /t REG_DWORD /d 1 /f
```
</small>
</small>
</small>
* Download and install Git from [git-for-windows.github.io](https://git-for-windows.github.io).
* Configure Git to accept long file names: `git config --system core.longpaths true`
* Download and install Python 3+ from [www.python.org/downloads/windows](https://www.python.org/downloads/windows).
* Don't forget to select option to add Python binaries to PATH environment!
* Configure Windows to accept long fie names during Python installation!

* Install CM via PIP:

```bash
python -m pip install cmind
```

*Note that we [have reports](https://github.com/mlcommons/ck/issues/844)
that CM does not work when Python was first installed from the Microsoft Store.
If CM fails to run, you can find a fix [here](https://stackoverflow.com/questions/57485491/python-python3-executes-in-command-prompt-but-does-not-run-correctly)*.


*We plan to provide a self-sustained package in the future to simplify CM installation on Windows.*



# CM CLI testing

If the installation is successful, you can run the CM CLI as follows:

```bash
gfursin@cmind:~$ cmx

cmx {action} {automation} {artifact(s)} {flags} @input.yaml @input.json
```

0 comments on commit 342cc44

Please sign in to comment.