Add a new `manifest` command to support multi-architecture builds #1705

husni-faiz · 2023-04-05T14:42:14Z

Hi all,

Multi-architecture support for buildpacks have been requested (#1459 | #1460 ) for a long time. Recently I have been working on this as part of the Linux Foundation Mentorship program under the mentorship of Juan Bustamante (@jjbustamante), Jerico Pena (@jericop) and Aidan Delaney (@AidanDelaney)

We would appreciate your ideas and thoughts on how we want to move forward. There is a lot of validations and test to be added along the line. We are having a weekly meetings for multi-arch support every Monday @ 10:00 AM EDT/EST. You can find the zoom link in the meeting minutes document.

A new ImageIndex/ManifestList interface is being added to imgutil module.

You may find the following resources useful.

Multi-Architecutre goals
Image Index specification
ImageIndex imgutil interface
Meeting minutes
Slack Discussion
pack manifest command
Pack issue Provide a way to specify desired platform when creating packages and builders #1459
Pack issue It would be nice to support easily creating a manifest list packages and builders #1460

Fixes #1678
Fixes #1460
Fixes #1722
Fixes #1720
Fixes #1721
Fixes #1719
Fixes #1718

jjbustamante

It is looking good, let's work on the unit testing and error handling

internal/commands/manifest_add.go

internal/commands/manifest_annotate.go

internal/commands/manifest_create.go

pkg/client/create_manifest.go

Add a new manifest command to pack root commands and add a create subcommand to the manifest command. Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

Use source/sinks instead of using primitives directly Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

Function templates for the following manifest sub command logics - create - add - annotate - push - remove Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

Command templates for the following manifest sub commands - Add - Annotate - Push - Remove Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

pkg/client/add_manifest.go

internal/commands/manifest_push_test.go

internal/commands/manifest_push.go

natalieparellano · 2024-05-01T15:10:47Z

Great work @husni-faiz @jjbustamante :) I left a few code suggestions and a few comments, but the only ones that are truly blocking for me are

Filename should be consistent with the command implemented
Let's decide on the user interface for media types (or at least explain what we mean by 'v2s2')
User interface for push options - ideally we only accept those when there is a --publish flag or we are calling push

Edit: on the last comment, I think that is actually what we are doing. I'll leave a code suggestion...

internal/commands/manifest_create.go

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

jjbustamante · 2024-05-02T15:18:51Z

@natalieparellano Thanks a lot for the feedback

Filename should be consistent with the command implemented

I just refactored the files

Let's decide on the user interface for media types (or at least explain what we mean by 'v2s2')

I changed it for oci or docker default value is oci

User interface for push options - ideally we only accept those when there is a --publish flag or we are calling push

Yeah, I think what is missing here is to add some validation to accept --insecure when --publish flag is used.

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

pkg/index/index_factory_test.go

natalieparellano · 2024-05-06T18:57:04Z

pkg/index/index_factory_test.go

+			it("finds the index in the remote registry", func() {
+				imageIndex, err = indexFactory.FindIndex(indexRepoName)
+				h.AssertNil(t, err)
+				h.AssertNotNil(t, imageIndex)


Do we want to assert anything about the index, such as its name?

We could, but we don't have any method to expose those attributes.

pkg/index/index_factory_test.go

…ndex Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

Implement image index interface in support of https://github.com/buildpacks/rfcs/blob/main/text/0124-pack-manifest-list-commands.md and buildpacks/pack#1705 Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com> Signed-off-by: Juan Bustamante <jbustamante@vmware.com> Signed-off-by: Natalie Arellano <narellano@vmware.com> Signed-off-by: Sai Kiran <sai90100msk@gmail.com> Signed-off-by: Sai Kiran Maggidi <107541780+WYGIN@users.noreply.github.com> Signed-off-by: WYGIN <sai90100msk@gmail.com> Signed-off-by: WYGIN <wygininc@gmail.com>

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

natalieparellano

LGTM (blocked on #2136)

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

Signed-off-by: Juan Bustamante <juan.bustamante@broadcom.com>

github-actions bot added this to the 0.30.0 milestone Apr 5, 2023

github-actions bot added type/chore Issue that requests non-user facing changes. type/enhancement Issue that requests a new feature or improvement. labels Apr 5, 2023

husni-faiz mentioned this pull request Apr 5, 2023

Manifest List Interface buildpacks/imgutil#191

Merged

jjbustamante reviewed May 3, 2023

View reviewed changes

internal/commands/manifest_add.go Outdated Show resolved Hide resolved

internal/commands/manifest_annotate.go Outdated Show resolved Hide resolved

internal/commands/manifest_create.go Outdated Show resolved Hide resolved

pkg/client/create_manifest.go Outdated Show resolved Hide resolved

husni-faiz force-pushed the test-manifest branch from 903e3fb to afc2669 Compare May 9, 2023 15:49

natalieparellano modified the milestones: 0.30.0, 0.31.0 May 19, 2023

github-actions bot modified the milestones: 0.31.0, 0.30.0 May 23, 2023

natalieparellano modified the milestones: 0.30.0, 0.31.0 May 30, 2023

github-actions bot modified the milestones: 0.31.0, 0.30.0 Jun 1, 2023

husni-faiz added 16 commits June 5, 2023 17:56

commands: manifest: command to handle manifests

835902d

Add a new manifest command to pack root commands and add a create subcommand to the manifest command. Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

cmd: manifest create: first PoC

ffe1f93

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

cmd: manifest: refactor code

08ec406

Use source/sinks instead of using primitives directly Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

cmd: manifest: use ggcr library

9b2a978

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

cmd: manifest: use imgutil

3fc79cc

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

manifest: use pack client interface

6b6f2b0

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

manifest_create: separate input validation and manifest creation

e3908c2

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

client: manifest sub command logic function templates

d767b1e

Function templates for the following manifest sub command logics - create - add - annotate - push - remove Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

commands: manifest sub command templates

3c0b9cc

Command templates for the following manifest sub commands - Add - Annotate - Push - Remove Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

bugfix: manifest: create: use single char for shorthand flag

e9ca7cb

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

manifest: create: move flag validations into command logic

3a7c262

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

manifest: create: remove path option from remote index:

40eade3

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

create_manifest: raise error if saving the index fails

67d5644

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

manifest_create: add default layout path to save an index

b6b0f0d

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

manifest: flag command as experimental

43f97b5

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

client: create an index factory

c3ac0ce

Signed-off-by: Husni Faiz <ahamedhusni73@gmail.com>

natalieparellano reviewed May 1, 2024

View reviewed changes

pkg/client/add_manifest.go Outdated Show resolved Hide resolved

natalieparellano reviewed May 1, 2024

View reviewed changes

internal/commands/manifest_push_test.go Outdated Show resolved Hide resolved

natalieparellano reviewed May 1, 2024

View reviewed changes

internal/commands/manifest_push.go Show resolved Hide resolved

natalieparellano reviewed May 1, 2024

View reviewed changes

internal/commands/manifest_create.go Outdated Show resolved Hide resolved

jjbustamante added 2 commits May 2, 2024 09:20

Fixing Natalie May 1 reviews comments

5f1a28c

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

fixing test

c7d4c06

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

jjbustamante and others added 9 commits May 2, 2024 10:46

adding client manifest push commands tests

c71db73

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

Merge branch 'main' into test-manifest

9ae682e

updating to docker 26 the last code from main

000fb60

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

organizing the manifest commands unit test

ca37f34

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

re-org the rm manifest tests

72ae7f2

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

Merge branch 'main' into test-manifest

b974278

simplifying the error handling on manifest delete/remove command

12cbadf

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

Merge branch 'main' into test-manifest

346fe1b

adding IndexFactory test coverage

b2ad463

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

natalieparellano reviewed May 6, 2024

View reviewed changes

pkg/index/index_factory_test.go Show resolved Hide resolved

natalieparellano reviewed May 6, 2024

View reviewed changes

pkg/index/index_factory_test.go Show resolved Hide resolved

natalieparellano reviewed May 6, 2024

View reviewed changes

pkg/index/index_factory_test.go Show resolved Hide resolved

natalieparellano reviewed May 6, 2024

View reviewed changes

pkg/index/index_factory_test.go Show resolved Hide resolved

Fixing an error when format is not specified when creating an image i…

6e7f0f2

…ndex Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

bumping imgutil version to the one with image index implementation

09a6373

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

natalieparellano approved these changes May 7, 2024

View reviewed changes

jjbustamante and others added 2 commits May 7, 2024 16:11

Adding acceptance test for manifest commands

922af8b

Signed-off-by: Juan Bustamante <jbustamante@vmware.com>

Merge branch 'main' into test-manifest

318ad0d

Signed-off-by: Juan Bustamante <juan.bustamante@broadcom.com>

jjbustamante approved these changes May 8, 2024

View reviewed changes

jjbustamante merged commit 67feb16 into buildpacks:main May 8, 2024
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a new `manifest` command to support multi-architecture builds #1705

Add a new `manifest` command to support multi-architecture builds #1705

husni-faiz commented Apr 5, 2023 •

edited by jjbustamante

Loading

jjbustamante left a comment

natalieparellano commented May 1, 2024 •

edited

Loading

jjbustamante commented May 2, 2024

natalieparellano May 6, 2024

jjbustamante May 6, 2024

natalieparellano left a comment

Add a new manifest command to support multi-architecture builds #1705

Add a new manifest command to support multi-architecture builds #1705

Conversation

husni-faiz commented Apr 5, 2023 • edited by jjbustamante Loading

jjbustamante left a comment

Choose a reason for hiding this comment

natalieparellano commented May 1, 2024 • edited Loading

jjbustamante commented May 2, 2024

natalieparellano May 6, 2024

Choose a reason for hiding this comment

jjbustamante May 6, 2024

Choose a reason for hiding this comment

natalieparellano left a comment

Choose a reason for hiding this comment

Add a new `manifest` command to support multi-architecture builds #1705

Add a new `manifest` command to support multi-architecture builds #1705

husni-faiz commented Apr 5, 2023 •

edited by jjbustamante

Loading

natalieparellano commented May 1, 2024 •

edited

Loading