Useful Commands

This README contains commands that I find useful, particularly on UNIX-like operating systems.

Table of Contents

• Conda
• Docker
• Git
• Python
• System Resources
• Miscellaneous

Conda

Create virtual environment

conda create -n <env_name> [<package>...] [-y]

<package>...: List of packages to install in the environment.

-y: Answer "yes" to any confirmations automatically.

Example:

conda create -n ml python=3.9 numpy -y

Enter virtual environment

conda activate <env_name>

Example:

conda activate ml

Exit virtual environment

conda deactivate <env_name>

Example:

conda deactivate ml

Clear conda cache

conda clean -a

Docker

Show all images

docker images

Show all containers

docker ps -a [--size]

--size: Show the size of each container.

Pull image

docker pull <repo>[:<tag>]

:<tag>: Specify tag <tag>, which defaults to latest, from repository <repo>.

Example:

docker pull nvidia/cuda:11.8.0-devel-ubuntu22.04

Create and run container

docker run \
    [--gpus all] \
    [-p <host_port>:<container_port>] \
    [-v <host_dir>:<container_dir>] \
    [{-d, -it}] \
    [--rm] \
    [--name <container_name>] \
    {<repo>[:<tag>], <image_id>} \
    [<arg>...]

--gpus all: Access all available GPUs.

-p <host_port>:<container_port>: Map <container_port> on the container to <host_port> on the host.

-v <host_dir>:<container_dir>: Mount <host_dir> on the host to <container_dir> inside the container.

-d: Run the container in background (i.e., detached mode).

-it: Keep STDIN open even if not attached (i.e., interactive mode).

--rm: Automatically remove the container when it exits.

--name <container_name>: Assign name <container_name> to the container.

<arg>...: Arguments passed to ENTRYPOINT (if any, in Dockerfile).

Example:

docker run -it --name deeplearning nvidia/cuda:11.8.0-devel-ubuntu22.04

Detach from container

Press Ctrl + p, then Ctrl + q.

Attach to running container

docker attach <container_name_or_id>

Example:

docker attach deeplearning

Stop container

docker {stop, kill} <container_name_or_id>

stop: Stop gracefully.

kill: Stop immediately.

Example:

docker stop deeplearning

Restart stopped container

docker start [-i] <container_name_or_id>

-i: Attach to container's STDIN (i.e., interactive mode).

Example:

docker start -i deeplearning

Copy files to container

docker cp <local_source> <container_name_or_id>:<container_destination>

Example:

docker cp ./training_scripts/ deeplearning:/

Copy files from container

docker cp <container_name_or_id>:<container_source> <local_destination>

Example:

docker cp deeplearning:/models/model_final.pth ./models/

Start new process inside container

docker exec -it <container_name_or_id> <command>

Example:

docker exec -it deeplearning /bin/bash

Remove container

docker rm <container_name_or_id>

Example:

docker rm deeplearning

Remove image

docker image rm {<repo>[:<tag>], <image_id>}

Build image from Dockerfile

docker build \
    [--platform <arch>] \
    [--build-arg <arg>=<value>] \
    [--no-cache] \
    [-t <repo>[:<tag>]] \
    <path/to/Dockerfile>

--platform <arch>: Set build platform/architecture to <arch>.

--build-arg <arg>=<value>: Set build-time variable <arg> to <value>. If you want to set multiple build-time variables, use multiple --build-args.

--no-cache: Do not use cache when building the image.

-t <repo>[:<tag>]: Assign repository name <repo> to the image. Optionally, assign tag <tag> to the image, which defaults to latest.

Example:

docker build -t my-image .

Save container as image

docker commit <container_name_or_id> [<repo>[:<tag>]]

Export image as .tar

docker save -o <output_file>.tar {<repo>[:tag], <image_id>}

Load image from .tar

docker load -i <input_file>.tar

Clear docker cache

docker builder prune [-a]

-a: Remove all unused build cache, not just dangling ones.

Git

Set commit email

git config [--global] user.email <email>

--global: Set the commit email for every repository on the computer.

Set commit username

git config [--global] user.name "<username>"

--global: Set the commit username for every repository on the computer.

Clone public repository

git clone {<repo_url>, <username>/<repo_name>}

Clone private repository

git clone https://<pat>@github.com/<username>/<repo_name>.git

For more information on how to generate Personal Access Token (PAT), visit How to clone a private repository from GitHub.

Show uncommitted changes

git status

Show differences

git diff

Add changes to staging area

git add <file>...

Example:

git add .

Restore files in working directory

git restore <file>...

Commit changes to local repository

git commit -m "<message>"

Push local changes to remote repository

git push

Pull remote changes to local repository

git pull

Python

Install Package

python -m pip install [--no-deps] \
    [--no-cache-dir] [-q] [-U] \
    {<package>..., -r <path/to/requirements>}

--no-deps: Do not install package dependencies.

--no-cache-dir: Disable caching.

-q: Give less output.

-U: Upgrade <package>... to the newest available version.

-r <path/to/requirements>: Install from <path/to/requirements>.

Install from public GitHub repository

python -m pip install git+https://github.com/<username>/<repo_name>.git[@<branch>]

@<branch>: Select desired branch.

Example:

python -m pip install git+https://github.com/anson416/python-utilities.git

Install from private GitHub repository

python -m pip install git+https://<pat>@github.com/<username>/<repo_name>.git[@<branch>]

Create requirements.py

python -m pip freeze > requirements.py

Clear pip cache

python -m pip cache purge

System Resources

Show running processes

top

Show info about CPU

lscpu

Show info about memory

free -h

Monitor CPU & memory

htop

Monitor NVIDIA GPU

watch -n 1 nvidia-smi

Show disk usage

df -h

Show directory size

du -h <dir>

Example:

du -h datasets/

Miscellaneous

Download files

wget [-P <dir>] <urls>

-P: Set download directory to <dir>, which defaults to . (i.e., current directory).

Zip files

zip -r ./<zip_name>.zip <file>... [-x <exclude>...]