Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: some improvements for usage #1442

Merged
merged 2 commits into from
Nov 9, 2023
Merged

docs: some improvements for usage #1442

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3900f25
docs: some improvements for usage
imeoer Oct 16, 2023
1838200
docs: unify object_prefix field for oss/s3 backend
imeoer Nov 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ The following Benchmarking results demonstrate that Nydus images significantly o
| Storage | Registry/OSS/S3/NAS | Support for OCI-compatible distribution implementations such as Docker Hub, Harbor, Github GHCR, Aliyun ACR, NAS, and Aliyun OSS-like object storage service | ✅ |
| Storage/Build | [Harbor](https://github.com/goharbor/acceleration-service) | Provides a general service for Harbor to support acceleration image conversion based on kinds of accelerator like Nydus and eStargz etc | ✅ |
| Distribution | [Dragonfly](https://github.com/dragonflyoss/Dragonfly2) | Improve the runtime performance of Nydus image even further with the Dragonfly P2P data distribution system | ✅ |
| Build | [Buildkit](https://github.com/moby/buildkit/blob/master/docs/nydus.md) | Provides the ability to build and export Nydus images directly from Dockerfile | ✅ |
| Build | [Buildkit](https://github.com/nydusaccelerator/buildkit/blob/master/docs/nydus.md) | Provides the ability to build and export Nydus images directly from Dockerfile | ✅ |
imeoer marked this conversation as resolved.
Show resolved Hide resolved
| Build/Runtime | [Nerdctl](https://github.com/containerd/nerdctl/blob/master/docs/nydus.md) | The containerd client to build or run (requires nydus snapshotter) Nydus image | ✅ |
| Runtime | [Docker / Moby](https://github.com/dragonflyoss/nydus/blob/master/docs/docker-env-setup.md) | Run Nydus image in Docker container with containerd and nydus-snapshotter | ✅ |
| Runtime | [Kubernetes](https://github.com/containerd/nydus-snapshotter/blob/main/docs/run_nydus_in_kubernetes.md) | Run Nydus image using CRI interface | ✅ |
Expand All @@ -89,7 +89,7 @@ make docker-static

Convert OCIv1 image to Nydus image: [Nydusify](./docs/nydusify.md), [Acceld](https://github.com/goharbor/acceleration-service) or [Nerdctl](https://github.com/containerd/nerdctl/blob/master/docs/nydus.md#build-nydus-image-using-nerdctl-image-convert).

Build Nydus image from Dockerfile directly: [Buildkit](https://github.com/moby/buildkit/blob/master/docs/nydus.md).
Build Nydus image from Dockerfile directly: [Buildkit](https://github.com/nydusaccelerator/buildkit/blob/master/docs/nydus.md).

Build Nydus layer from various sources: [Nydus Image Builder](./docs/nydus-image.md).

Expand Down
87 changes: 64 additions & 23 deletions docs/containerd-env-setup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,26 @@ sudo install -D -m 755 nydusd nydus-image nydusify nydusctl nydus-overlayfs /usr
sudo install -D -m 755 containerd-nydus-grpc /usr/bin
```

## Start a Local Registry Container

To make it easier to convert and run nydus images next, we can run a local registry service with docker:

```bash
sudo docker run -d --restart=always -p 5000:5000 registry
```

## Convert/Build an Image to Nydus Format

Nydus image can be created by converting from an existing OCI or docker v2 image stored in container registry or directly built from Dockerfile(with [Buildkit](https://github.com/nydusaccelerator/buildkit/blob/master/docs/nydus.md))

Note: For private registry repo, please make sure you are authorized to pull and push the target registry. The basic method is to use `docker pull` and `docker push` to verify your access to the source or target registry.

```bash
sudo nydusify convert --source ubuntu --target localhost:5000/ubuntu-nydus
```

For more details about how to build nydus image, please refer to [Nydusify](https://github.com/dragonflyoss/nydus/blob/master/docs/nydusify.md) conversion tool, [Acceld](https://github.com/goharbor/acceleration-service) conversion service or [Nerdctl](https://github.com/containerd/nerdctl/blob/master/docs/nydus.md#build-nydus-image-using-nerdctl-image-convert).

## Start Nydus Snapshotter

Nydus provides a containerd remote snapshotter `containerd-nydus-grpc` (nydus snapshotter) to prepare container rootfs with nydus formatted images.
Expand Down Expand Up @@ -82,7 +102,7 @@ sudo /usr/bin/containerd-nydus-grpc \
--log-to-stdout
```

## Configure and Start Containerd
## [Option 1] Configure as Containerd Global Snapshotter

Nydus depends on two features of Containerd:

Expand All @@ -107,6 +127,7 @@ For version 1 containerd config format:
[plugins.cri.containerd]
snapshotter = "nydus"
disable_snapshot_annotations = false
discard_unpacked_layers = false
```

For version 2 containerd config format:
Expand All @@ -115,6 +136,7 @@ For version 2 containerd config format:
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "nydus"
disable_snapshot_annotations = false
discard_unpacked_layers = false
```

Then restart containerd, e.g.:
Expand All @@ -123,19 +145,45 @@ Then restart containerd, e.g.:
sudo systemctl restart containerd
```

### Using Runtime-level Snapshotter in Container for Nydus
Containerd (>= v1.7.0) supports configuring the ```runtime-level``` snapshotter. By following the steps below, we can declare runtimes that use different snapshotters:
#### Step 1: Configure Containerd
## [Option 2] Configure as Containerd Runtime-Level Snapshotter

Note: this way only works on CRI based scenario (for example crictl or kubernetes).

Containerd (>= v1.7.0) supports configuring the `runtime-level` snapshotter. By following the steps below, we can declare runtimes that use different snapshotters:
imeoer marked this conversation as resolved.
Show resolved Hide resolved

### Step 1: Apply Containerd Patches

[Patch](https://github.com/nydusaccelerator/containerd/commit/0959cdb0b190e35c058a0e5bc2e256e59b95b584): fixes the handle of sandbox run and container create for runtime-level snapshotter;

### Step 2: Configure Containerd

Only for version 2 containerd config format:

```toml
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "overlayfs"
disable_snapshot_annotations = false
discard_unpacked_layers = false
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nydus-runc]

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc-nydus]
snapshotter = "nydus"

[proxy_plugins]
[proxy_plugins.nydus]
type = "snapshot"
address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
```

Then restart containerd, e.g.:

```bash
sudo systemctl restart containerd
```
#### Step 2: Add an Extra Annotation in Sandbox Spec
The annotation ```"io.containerd.cri.runtime-handler": "nydus-runc"``` must be set in sandbox spec. The ```nydus-sandbox.yaml``` looks like below:

### Step 3: Add an Extra Annotation in Sandbox Spec

The annotation `"io.containerd.cri.runtime-handler": "runc-nydus"` must be set in sandbox spec. The `nydus-sandbox.yaml` looks like below:

```yaml
metadata:
attempt: 1
Expand All @@ -147,29 +195,24 @@ linux:
namespace_options:
network: 2
annotations:
"io.containerd.cri.runtime-handler": "nydus-runc"
"io.containerd.cri.runtime-handler": "runc-nydus"
```
As shown above, the sandbox is declared with ```"io.containerd.cri.runtime-handler": "nydus-runc"``` annotation will use the ```nydus``` snapshotter, while others will use the default ```overlayfs``` snapshotter.

## Start a Local Registry Container
As shown above, the sandbox is declared with `"io.containerd.cri.runtime-handler": "runc-nydus"` annotation will use the `nydus` snapshotter, while others will use the default `overlayfs` snapshotter.

To make it easier to convert and run nydus images next, we can run a local registry service with docker:
Note: You may encounter the following error when creating a Pod:

```bash
sudo docker run -d --restart=always -p 5000:5000 registry
```
err="failed to \"StartContainer\" for \"xxx\" with CreateContainerError: \"failed to create containerd container: error unpacking image: failed to extract layer sha256:yyy: failed to get reader from content store: content digest sha256:zzz: not found\""
```

## Convert/Build an Image to Nydus Format

Nydus image can be created by converting from an existing OCI or docker v2 image stored in container registry or directly built from Dockerfile(with [Buildkit](https://github.com/moby/buildkit/blob/master/docs/nydus.md))
This is because some images in the Pod (including the Pause image) have used containerd's default snapshotter (such as the `overlayfs`` snapshotter), and the `discard_unpacked_layers` option was previously set to `true`, containerd has already deleted the blobs from the content store. To resolve this issue, you should first ensure that `discard_unpacked_layers=false`, then use the following command to restore the image:

Note: For private registry repo, please make sure you are authorized to pull and push the target registry. The basic method is to use `docker pull` and `docker push` to verify your access to the source or target registry.

```bash
sudo nydusify convert --source ubuntu --target localhost:5000/ubuntu-nydus
```
ctr -n k8s.io content fetch pause:3.8
```

For more details about how to build nydus image, please refer to [Nydusify](https://github.com/dragonflyoss/nydus/blob/master/docs/nydusify.md) conversion tool, [Acceld](https://github.com/goharbor/acceleration-service) conversion service or [Nerdctl](https://github.com/containerd/nerdctl/blob/master/docs/nydus.md#build-nydus-image-using-nerdctl-image-convert).
Please note that `pause:3.8` is just an example image, you should also fetch all images used by the Pod to ensure that there are no issues.

## Try Nydus with `nerdctl`

Expand All @@ -195,8 +238,6 @@ linux:
security_context:
namespace_options:
network: 2
annotations:
"io.containerd.osfeature": "nydus.remoteimage.v1"
```

The `nydus-container.yaml` looks like below:
Expand Down
4 changes: 3 additions & 1 deletion docs/nydus-image.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,9 +192,11 @@ data blobs: ["9e50ae5ac02b2ef6ffb86075720e49d95d8240eed4717dd8ac9c68cadba00762"]
cat /path/to/backend-config.json
{
"endpoint": "region.aliyuncs.com",
"scheme": "https",
"access_key_id": "",
"access_key_secret": "",
"bucket_name": ""
"bucket_name": "",
"object_prefix": "nydus/"
}

# min_used_ratio:
Expand Down
137 changes: 83 additions & 54 deletions docs/nydusd.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,8 @@ We are working on enabling cloud-hypervisor support for nydus.

#### Use Different Storage Backends

Using different storage backend means that the nydus image metadata (bootstrap) layer is stored in the image registry, but the data layer will be stored on the external storage. Therefore, the option `--target` for `nydusify convert` is still required, the registry image reference is needed to store the metadata layer.

##### Localfs Backend

```
Expand Down Expand Up @@ -192,7 +194,7 @@ Document located at: https://github.com/adamqqqplay/nydus-localdisk/blob/master/
}
```

##### OSS backend with blobcache
##### OSS Backend

```
{
Expand All @@ -202,9 +204,35 @@ Document located at: https://github.com/adamqqqplay/nydus-localdisk/blob/master/
"config": {
...
"endpoint": "region.aliyuncs.com",
"scheme": "https",
"access_key_id": "",
"access_key_secret": "",
"bucket_name": "",
"object_prefix": "nydus/"
bergwolf marked this conversation as resolved.
Show resolved Hide resolved
}
},
...
},
...
}
```

##### S3 Backend

```
{
"device": {
"backend": {
"type": "s3",
"config": {
...
"endpoint": "s3.amazonaws.com",
"scheme": "https",
"access_key_id": "",
"access_key_secret": "",
"bucket_name": ""
"bucket_name": "",
"region": "",
"object_prefix": "nydus/"
}
},
...
Expand All @@ -213,7 +241,7 @@ Document located at: https://github.com/adamqqqplay/nydus-localdisk/blob/master/
}
```

##### Registry backend
##### Registry Backend

```
{
Expand Down Expand Up @@ -246,41 +274,51 @@ Document located at: https://github.com/adamqqqplay/nydus-localdisk/blob/master/
```
Note: The value of `device.backend.config.auth` will be overwrite if running the nydusd with environment variable `IMAGE_PULL_AUTH`.

##### Enable P2P Proxy for Storage Backend
#### HTTP Proxy Backend

Add `device.backend.config.proxy` field to enable HTTP proxy for storage backend. For example, use P2P distribution service to reduce network workload and latency in large scale container cluster using [Dragonfly](https://d7y.io/) (enable centralized dfdaemon mode).
The `HttpProxy` backend can access blobs through a http proxy server which can be local (using unix socket) or remote (using `https://` or using `http://`).

`HttpProxy` uses two API endpoints to access the blobs:
- `HEAD /path/to/blobs` to get the blob size
- `GET /path/to/blobs` to read the blob

The http proxy server should respect [the `Range` header](https://www.rfc-editor.org/rfc/rfc9110.html#name-range) to compute the offset and length of the blob.

The example config files for the `HttpProxy` backend may be:

```
// for remote usage
{
"device": {
"backend": {
"type": "registry",
"backend": {
"type": "http-proxy",
"config": {
"proxy": {
// Access remote storage backend via P2P proxy, e.g. Dragonfly dfdaemon server URL
"url": "http://p2p-proxy:65001",
// Fallback to remote storage backend if P2P proxy ping failed
"fallback": true,
// Endpoint of P2P proxy health checking
"ping_url": "http://p2p-proxy:40901/server/ping",
// Interval of P2P proxy health checking, in seconds
"check_interval": 5
},
...
"addr": "http://127.0.0.1:9977",
"path": "/namespace/<repo>/blobs"
}
},
...
},
...
}
}
}
```

Once the configuration is loaded successfully on nydusd starting, we will see the log as shown below:
or

```
INFO [storage/src/backend/connection.rs:136] backend config: CommonConfig { proxy: ProxyConfig { url: "http://p2p-proxy:65001", ping_url: "http://p2p-proxy:40901/server/ping", fallback: true, check_interval: 5 }, timeout: 5, connect_timeout: 5, retry_limit: 0 }
// for local usage
{
"device": {
"backend": {
"type": "http-proxy",
"config": {
"addr": "/path/to/unix.sock",
}
}
}
}
```

The `HttpProxy` backend also supports the `Proxy` and `Mirrors` configurations for remote usage like the `Registry backend` described above.

##### Enable Mirrors for Storage Backend (Recommend)

Nydus is deeply integrated with [Dragonfly](https://d7y.io/) P2P mirror mode, please refer the [doc](https://d7y.io/docs/setup/integration/nydus) to learn how configuring Nydus to use Dragonfly.
Expand Down Expand Up @@ -332,51 +370,42 @@ Currently, the mirror mode is only tested in the registry backend, and in theory
}
```

#### HTTP proxy backend

The `HttpProxy` backend can access blobs through a http proxy server which can be local (using unix socket) or remote (using `https://` or using `http://`).

`HttpProxy` uses two API endpoints to access the blobs:
- `HEAD /path/to/blobs` to get the blob size
- `GET /path/to/blobs` to read the blob

The http proxy server should respect [the `Range` header](https://www.rfc-editor.org/rfc/rfc9110.html#name-range) to compute the offset and length of the blob.
##### Enable P2P Proxy for Storage Backend

The example config files for the `HttpProxy` backend may be:
Add `device.backend.config.proxy` field to enable HTTP proxy for storage backend. For example, use P2P distribution service to reduce network workload and latency in large scale container cluster using [Dragonfly](https://d7y.io/) (enable centralized dfdaemon mode).

```
// for remote usage
{
"device": {
"backend": {
"type": "http-proxy",
"backend": {
"type": "registry",
"config": {
"addr": "http://127.0.0.1:9977",
"path": "/namespace/<repo>/blobs"
"proxy": {
// Access remote storage backend via P2P proxy, e.g. Dragonfly dfdaemon server URL
"url": "http://p2p-proxy:65001",
// Fallback to remote storage backend if P2P proxy ping failed
"fallback": true,
// Endpoint of P2P proxy health checking
"ping_url": "http://p2p-proxy:40901/server/ping",
// Interval of P2P proxy health checking, in seconds
"check_interval": 5
},
...
}
}
}
},
...
},
...
}
```

or
Once the configuration is loaded successfully on nydusd starting, we will see the log as shown below:

```
// for local usage
{
"device": {
"backend": {
"type": "http-proxy",
"config": {
"addr": "/path/to/unix.sock",
}
}
}
}
INFO [storage/src/backend/connection.rs:136] backend config: CommonConfig { proxy: ProxyConfig { url: "http://p2p-proxy:65001", ping_url: "http://p2p-proxy:40901/server/ping", fallback: true, check_interval: 5 }, timeout: 5, connect_timeout: 5, retry_limit: 0 }
```

The `HttpProxy` backend also supports the `Proxy` and `Mirrors` configurations for remote usage like the `Registry backend` described above.

### Mount Bootstrap Via API

To mount a bootstrap via api, first launch nydusd without a bootstrap:
Expand Down
Loading