Bundle a buildkit instance within sandbox to facilitate local image spec builds

[jeevb]

Related: flyteorg/flytectl#413

Tested by running the following command:

envd context create --name flyte-sandbox --builder tcp --builder-address localhost:30003 --use

and executing (pyflyte run --remote wf.py my_wf) the following workflow:

from flytekit import task, workflow, ImageSpec

image = ImageSpec(
    name="imagespec",
    registry="localhost:30000",
    packages=["smart-open[gcs]"],
    apt_packages=["git"],
)


@task(container_image=image)
def my_task() -> str:
    return "hello, world!"


@workflow
def my_wf() -> str:
    return my_task()

[jeevb]

@eapolinario @cosmicBboy @wild-endeavor: For discussion.

[wild-endeavor]

can you explain what's happening? what's the interaction between envd and buildkit? I don't remember having to run a buildkit container when running envd outside of the sandbox. why do i need it now when running something inside?

[jeevb]

can you explain what's happening? what's the interaction between envd and buildkit? I don't remember having to run a buildkit container when running envd outside of the sandbox. why do i need it now when running something inside?

envd uses buildkit under the hood to build images. If you run envd context ls, you can see that it defaults to using a local docker container for running builds. This container is created lazily when processing builds. The issue with using a separate container is that docker networking is not set up to allow this container to talk to the registry at localhost:30000. So it basically can't read the registry to check for the presence of an existing image (or push a new image to it). Running buildkit inside the sandbox allows us to fully manage the networking so that the experience with using ImageSpec locally is seamless, with a small cost of ~165MB.