Add possibility to download IPFS images #2408

afbjorklund · 2024-06-09T11:00:17Z

See https://docs.ipfs.tech/how-to/kubo-basic-cli/ for ipfs

https://docs.ipfs.tech/how-to/address-ipfs-on-web/#native-urls

ipfs://{cidv1}/path/to/resource (see also https://cid.ipfs.tech/)

afbjorklund · 2024-06-09T11:16:50Z

The lima.yaml would then look something like:

- location: "ipfs://QmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK/ubuntu-24.04-server-cloudimg-amd64.img"
  arch: "x86_64"
  digest: "sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3"

(the file name is only used for decompression)

Note that the CID digest is not the file digest:

https://cid.ipfs.tech/#QmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK
6278B63498EB92816C50A53202EE3CBEE6FC0F92F97B97CB0AB0A4AE65CCBE38

https://docs.ipfs.tech/concepts/content-addressing/#cids-are-not-file-hashes

(small detail: zQmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK in multihash format is the same as
"sha256:6278b63498eb92816c50a53202ee3cbee6fc0f92f97b97cb0ab0a4ae65ccbe38" in text format)

afbjorklund · 2024-06-09T11:22:57Z

Note: the ipfs tool will output v0 by default, unless using --cid-version 1

v0: QmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK
v1: bafybeig5sch22ecfox7gq724rz7uivydwvnnpuqdcnjz72iwelgtrakzui

$ ipfs add --cid-version 0 ubuntu-24.04-server-cloudimg-amd64.img 
added QmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK ubuntu-24.04-server-cloudimg-amd64.img
 453.00 MiB / 453.00 MiB [=================================================================================================================] 100.00%
$ ipfs add --cid-version 1 ubuntu-24.04-server-cloudimg-amd64.img 
added bafybeig5sch22ecfox7gq724rz7uivydwvnnpuqdcnjz72iwelgtrakzui ubuntu-24.04-server-cloudimg-amd64.img
 453.00 MiB / 453.00 MiB [=================================================================================================================] 100.00%

The CID version doesn't really matter to Lima, but CIDv0 is deprecated.

https://docs.ipfs.tech/concepts/content-addressing/#cid-versions

afbjorklund · 2024-06-10T10:01:07Z

Using ipfs cat to get our own progress, and ipfs ls to calculate the size.

Hash                                                        Size     Name
bafybeibnerap2c5tnmqvyftmyxftftjewvo52acwv2gg6thpipyw5zx7fe 45613056 
bafybeidrbml5blbqj2i67x5gujqbwluis6jy5daiso6ir5pakx4hseztum 45613056 
bafybeie56ruaziz43jj3e5iih5r3fyzudeurqbpts6vwigf7nbreoep5de 45613056 
bafybeifva7p55d4szahrrxqjdjqqqlcrcfxm7n2vnpmwucrdkvcexxngmu 45613056 
bafybeie5wduudyjkbcyjvcif7i6gliipapvl5cw4wia3fqoanxunud7dyi 45613056 
bafybeidezz5b2zng3etcwbn4vnu3tn4ao3wrh5qdwzoyo7ywdp7rbymgzq 45613056 
bafybeiaf7vcteys366fmjiclktnzwuqpygpn4u2ftqrl62juaq5sgjw4be 45613056 
bafybeiasvhz6mljylbqwygb7jvp6d6sohaw325yuan5awlo7shjrlslb7m 45613056 
bafybeieiuxw2i5nkelq5tihd2tgmtxjst2ln6rslt35jxslcz4w4mjsnha 45613056 
bafybeiest5wjnmjwyp7vgawfbityfnkgqpr27bfleysbhdrhldcwxw4yq4 45613056 
bafybeif7btzsv7657kfxu7qte6sh6zgpvpsrj7dnou7usj2nwv4zf77ft4 18874368

So now IPFS address looks the same as HTTP address, with "description":

Downloading the image (ubuntu-24.04-server-cloudimg-amd64.img)
453.00 MiB / 453.00 MiB [---------------------------------] 100.00% 336.20 MiB/s

Instead of the output that you get from ipfs get, that also could change.

Saving file(s) to /home/anders/.cache/lima/download/by-url-sha256/1855c5dccbd6db83ea6c81c276e0440ad9f156584e2ced824290186f1dae563b/data
 453.00 MiB / 453.00 MiB [==================================================================================] 100.00% 0s

afbjorklund · 2024-06-10T10:04:41Z

At first I was thinking that calculating the digest was unnecessary, since it already has one included in the storage.

But we still want to compare the download with the digest we are expecting, to make sure it's the same image...

AkihiroSuda · 2024-06-19T05:27:04Z

Looks good, but needs a documentation

afbjorklund · 2024-06-20T14:00:23Z

There is a design flaw with this approach. Currently it would look like:

- location: "https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-amd64.img"
  arch: "x86_64"
  digest: "sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3"
- location: "ipfs://QmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK/ubuntu-24.04-server-cloudimg-amd64.img"
  arch: "x86_64"
  digest: "sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3"

That means the checksum is duplicated, between the transports. Maybe:

- location: "https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-amd64.img"
  arch: "x86_64"
  digest: "sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3"
  cid: QmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK

Note:

The CID can be calculated, without adding the image to the disk store:

$ sha256sum ubuntu-24.04-server-cloudimg-amd64.img 
32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3  ubuntu-24.04-server-cloudimg-amd64.img
$ ipfs add --only-hash --quieter ubuntu-24.04-server-cloudimg-amd64.img 
QmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK

afbjorklund · 2024-07-29T08:33:50Z

Unfortunately, Content-Type and Last-Modified are not provided by IPFS...

Current gateway only use heuristics like file magic and relative freshness.

Note: new implementations are supposed to use CID version 1 (not version 0):

$ ipfs add --cid-version=1 --only-hash --quieter ubuntu-24.04-server-cloudimg-amd64.img 
bafybeig5sch22ecfox7gq724rz7uivydwvnnpuqdcnjz72iwelgtrakzui

afbjorklund · 2024-09-28T12:29:32Z

Looks good, but needs a documentation

Currently it assumes that IPFS Kubo is set up.

i.e. that ipfs add and ipfs get is working

https://docs.ipfs.tech/how-to/kubo-basic-cli/

This might also want to mention some experimental features, like using private networks:

https://github.com/ipfs/kubo/blob/v0.30.0/docs/experimental-features.md#private-networks

For testing purposes, you can use ipfs daemon --offline to avoid connecting to the swarm.

See also docs at: https://github.com/containerd/stargz-snapshotter/blob/main/docs/ipfs.md

pkg/limayaml/containerd.yaml

afbjorklund · 2024-10-09T09:04:11Z

Could also add support for IPFS_GATEWAY, as a fallback?

https://blog.ipfs.tech/ipfs-uri-support-in-curl/

If set, it would rewrite any ipfs: into http/https instead...

export IPFS_GATEWAY="http://127.0.0.1:8080"

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

AkihiroSuda · 2024-10-09T15:34:14Z

examples/default.yaml

@@ -299,6 +299,10 @@ rosetta:
 # 🟢 Builtin default: use name from /etc/timezone or deduce from symlink target of /etc/localtime
 timezone: null

+# Allow using IPFS for downloading files with a provided CID.
+# 🟢 Builtin default: false
+ipfs: null


Can we just use location: cid://<CID> ?

Or location: ipfs://<CID> ?

You can use ipfs: instead of https:, but then it becomes a different URL...

(which is bad because it would require multiple digests, and cache entries)

#2408 (comment)

Also it would make it mandatory to use ipfs (or a ipfs gateway), to download?

By having it as an attribute on the File, it is possible to have it optional (and share digest)

That means the checksum is duplicated, between the transports.

I think this is better than introducing more YAML fields?

Also it would make it mandatory to use ipfs (or a ipfs gateway), to download?

The downloader should just fall back to the next candidate in the []Images

It is possible to do this with the current implementation, i.e. it supports both ways

You can change the location (url), or you can provide a cid for an existing location

Since the fields are optional, they should not interfer so much with existing templates?

Probably want to have some more automation / sharing in place, before adding them.

afbjorklund · 2024-10-09T16:19:21Z

Regarding addressing of ipfs objects, there are some more details here:
https://github.com/ipfs/in-web-browsers/blob/master/ADDRESSING.md

"The four stages of the upgrade path for path addressing."

Current: HTTP-to-IPFS gateway
e.g. https://ipfs.io/ipfs/$hash
Short term: URL
e.g. ipfs://$hash
Mid term: URI
e.g. dweb:/ipfs/$hash
Long term: NURI
e.g. /ipfs/$hash

So it is simpler to only provide the CID/hash, as additional information?
https://docs.ipfs.tech/concepts/content-addressing/#what-is-a-cid

Since it is a multiformat/multihash, it doesn't need an additional prefix*.

* it actually already has a couple of them, but in a string encoded form
(one can use ipfs cid format, or https://cid.ipfs.io/, to decipher them)

cid: bafybeieipdaxd3fzy3j7syzzxdaqxramk65j7ajzcqgmi6b5jyq4jgbwue
# base32-cidv1-dag-pb-(sha2-256:32:8878C171ECB9C6D3F96339B8C10BC40C57BA9F8139140CC4783D4E21C49836A1)

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

afbjorklund · 2024-10-09T18:10:02Z

More user-facing documentation (for the website) can go in a second PR. Maybe just use the links above?

https://docs.ipfs.tech/how-to/kubo-basic-cli/

Also needs documentation on how to update images, then again we don't have any docs for sha256 either...

- location: https://github.com/containerd/nerdctl/releases/download/v1.7.6/nerdctl-full-1.7.6-linux-amd64.tar.gz
  arch: x86_64
  digest: sha256:2c841e097fcfb5a1760bd354b3778cb695b44cd01f9f271c17507dc4a0b25606
  cid: bafybeieipdaxd3fzy3j7syzzxdaqxramk65j7ajzcqgmi6b5jyq4jgbwue

Like, when updated to 1.7.7 - how do you update the other fields?

$ wget https://github.com/containerd/nerdctl/releases/download/v1.7.7/nerdctl-full-1.7.7-linux-amd64.tar.gz
...
HTTP request sent, awaiting response... 200 OK
Length: 259844835 (248M) [application/octet-stream]
Saving to: ‘nerdctl-full-1.7.7-linux-amd64.tar.gz’
$ sha256sum nerdctl-full-1.7.7-linux-amd64.tar.gz
a731eac93e8e9dda1a0d76dc1606438deb0668ea7d6bd5c5af436353ed9f65c5  nerdctl-full-1.7.7-linux-amd64.tar.gz
$ ipfs add --only-hash --cid-version=1 --progress=false nerdctl-full-1.7.7-linux-amd64.tar.gz 
added bafybeiexmdvas4d3dy3npvecj3udihifaqndhelpiyjb67zbsm3g5eqlba nerdctl-full-1.7.7-linux-amd64.tar.gz

afbjorklund force-pushed the ipfs branch from baf63db to 1b3d76b Compare June 23, 2024 15:57

afbjorklund marked this pull request as draft June 23, 2024 16:18

afbjorklund force-pushed the ipfs branch from 1b3d76b to 88c33a3 Compare July 29, 2024 08:28

afbjorklund force-pushed the ipfs branch 2 times, most recently from 41dc704 to 68fadb5 Compare September 28, 2024 08:37

afbjorklund marked this pull request as ready for review September 28, 2024 08:37

AkihiroSuda reviewed Oct 9, 2024

View reviewed changes

pkg/limayaml/containerd.yaml Show resolved Hide resolved

afbjorklund added 7 commits October 9, 2024 14:14

Add possibility to download IPFS images

d5d9626

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Refactor: download separate from http

c9b2df3

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Use internal progress also for ipfs

174457a

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Show stderr from the ipfs commands

3ec88c4

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Allow adding CID to existing files

cd7737e

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Add cid for containerd archives

a5b048a

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

Allow using an http ipfs gateway

fddec51

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

afbjorklund force-pushed the ipfs branch 3 times, most recently from be4e62f to f63bd86 Compare October 9, 2024 12:26

AkihiroSuda reviewed Oct 9, 2024

View reviewed changes

Add config for enabling IPFS

46b7446

Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com>

afbjorklund force-pushed the ipfs branch from f63bd86 to 46b7446 Compare October 9, 2024 18:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add possibility to download IPFS images #2408

Add possibility to download IPFS images #2408

afbjorklund commented Jun 9, 2024 •

edited

Loading

afbjorklund commented Jun 9, 2024 •

edited

Loading

afbjorklund commented Jun 9, 2024 •

edited

Loading

afbjorklund commented Jun 10, 2024 •

edited

Loading

afbjorklund commented Jun 10, 2024

AkihiroSuda commented Jun 19, 2024

afbjorklund commented Jun 20, 2024

afbjorklund commented Jul 29, 2024 •

edited

Loading

afbjorklund commented Sep 28, 2024 •

edited

Loading

afbjorklund commented Oct 9, 2024

AkihiroSuda Oct 9, 2024

AkihiroSuda Oct 9, 2024

afbjorklund Oct 9, 2024 •

edited

Loading

AkihiroSuda Oct 9, 2024

afbjorklund Oct 9, 2024

afbjorklund Oct 9, 2024

afbjorklund commented Oct 9, 2024 •

edited

Loading

afbjorklund commented Oct 9, 2024 •

edited

Loading

Add possibility to download IPFS images #2408

Are you sure you want to change the base?

Add possibility to download IPFS images #2408

Conversation

afbjorklund commented Jun 9, 2024 • edited Loading

afbjorklund commented Jun 9, 2024 • edited Loading

afbjorklund commented Jun 9, 2024 • edited Loading

afbjorklund commented Jun 10, 2024 • edited Loading

afbjorklund commented Jun 10, 2024

AkihiroSuda commented Jun 19, 2024

afbjorklund commented Jun 20, 2024

afbjorklund commented Jul 29, 2024 • edited Loading

afbjorklund commented Sep 28, 2024 • edited Loading

afbjorklund commented Oct 9, 2024

AkihiroSuda Oct 9, 2024

Choose a reason for hiding this comment

AkihiroSuda Oct 9, 2024

Choose a reason for hiding this comment

afbjorklund Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

AkihiroSuda Oct 9, 2024

Choose a reason for hiding this comment

afbjorklund Oct 9, 2024

Choose a reason for hiding this comment

afbjorklund Oct 9, 2024

Choose a reason for hiding this comment

afbjorklund commented Oct 9, 2024 • edited Loading

afbjorklund commented Oct 9, 2024 • edited Loading

afbjorklund commented Jun 9, 2024 •

edited

Loading

afbjorklund commented Jun 9, 2024 •

edited

Loading

afbjorklund commented Jun 9, 2024 •

edited

Loading

afbjorklund commented Jun 10, 2024 •

edited

Loading

afbjorklund commented Jul 29, 2024 •

edited

Loading

afbjorklund commented Sep 28, 2024 •

edited

Loading

afbjorklund Oct 9, 2024 •

edited

Loading

afbjorklund commented Oct 9, 2024 •

edited

Loading

afbjorklund commented Oct 9, 2024 •

edited

Loading