Running KFP v2 with non rooted image fails (using artefacts in pipeline) #2573

szymek116 · 2023-11-21T12:34:14Z

/kind bug
What steps did you take and what happened:

Generate KF pipeline with KFP SDK as below

from kfp import dsl
from kfp.dsl import (component, Output, ClassificationMetrics, Metrics, HTML,
                     Markdown)

@dsl.component
def say_hello(name: str) -> str:
    hello_text = f'Hello, {name}!'
    print(hello_text)
    return hello_text

@dsl.component
def html_visualization(html_artifact: Output[HTML]):
    html_content = '<!DOCTYPE html><html><body><h1>Hello world</h1></body></html>'
    with open(html_artifact.path, 'w') as f:
        f.write(html_content)

@dsl.pipeline
def hello_pipeline(recipient: str) -> str:
    hello_task = say_hello(name=recipient)
    html_visualization()
    return hello_task.output

from kfp import compiler

compiler.Compiler().compile(hello_pipeline, 'pipeline.yaml')

2.Create non rooted base image:

docker@minikube:~$ cat dockerfile 
FROM python:3.7
RUN useradd -m ttt
USER ttt

tagged here as python:3.7-mod

Replace in pipeline.yaml generate in point 1 image on the one generate in point 2, output yaml looks like below:

# PIPELINE DEFINITION
# Name: hello-pipeline
# Inputs:
#    recipient: str
# Outputs:
#    Output: str
components:
  comp-html-visualization:
    executorLabel: exec-html-visualization
    outputDefinitions:
      artifacts:
        html_artifact:
          artifactType:
            schemaTitle: system.HTML
            schemaVersion: 0.0.1
  comp-say-hello:
    executorLabel: exec-say-hello
    inputDefinitions:
      parameters:
        name:
          parameterType: STRING
    outputDefinitions:
      parameters:
        Output:
          parameterType: STRING
deploymentSpec:
  executors:
    exec-html-visualization:
      container:
        args:
        - --executor_input
        - '{{$}}'
        - --function_to_execute
        - html_visualization
        command:
        - sh
        - -c
        - "\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip ||\
          \ python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1\
          \ python3 -m pip install --quiet --no-warn-script-location 'kfp==2.4.0'\
          \ '--no-deps' 'typing-extensions>=3.7.4,<5; python_version<\"3.9\"' && \"\
          $0\" \"$@\"\n"
        - sh
        - -ec
        - 'program_path=$(mktemp -d)


          printf "%s" "$0" > "$program_path/ephemeral_component.py"

          _KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         "$program_path/ephemeral_component.py"                         "$@"

          '
        - "\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import\
          \ *\n\ndef html_visualization(html_artifact: Output[HTML]):\n    html_content\
          \ = '<!DOCTYPE html><html><body><h1>Hello world</h1></body></html>'\n  \
          \  with open(html_artifact.path, 'w') as f:\n        f.write(html_content)\n\
          \n"
        image: python:3.7-mod
    exec-say-hello:
      container:
        args:
        - --executor_input
        - '{{$}}'
        - --function_to_execute
        - say_hello
        command:
        - sh
        - -c
        - "\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip ||\
          \ python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1\
          \ python3 -m pip install --quiet --no-warn-script-location 'kfp==2.4.0'\
          \ '--no-deps' 'typing-extensions>=3.7.4,<5; python_version<\"3.9\"' && \"\
          $0\" \"$@\"\n"
        - sh
        - -ec
        - 'program_path=$(mktemp -d)


          printf "%s" "$0" > "$program_path/ephemeral_component.py"

          _KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         "$program_path/ephemeral_component.py"                         "$@"

          '
        - "\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import\
          \ *\n\ndef say_hello(name: str) -> str:\n    hello_text = f'Hello, {name}!'\n\
          \    print(hello_text)\n    return hello_text\n\n"
        image: python:3.7-mod
pipelineInfo:
  name: hello-pipeline
root:
  dag:
    outputs:
      parameters:
        Output:
          valueFromParameter:
            outputParameterKey: Output
            producerSubtask: say-hello
    tasks:
      html-visualization:
        cachingOptions:
          enableCache: true
        componentRef:
          name: comp-html-visualization
        taskInfo:
          name: html-visualization
      say-hello:
        cachingOptions:
          enableCache: true
        componentRef:
          name: comp-say-hello
        inputs:
          parameters:
            name:
              componentInputParameter: recipient
        taskInfo:
          name: say-hello
  inputDefinitions:
    parameters:
      recipient:
        parameterType: STRING
  outputDefinitions:
    parameters:
      Output:
        parameterType: STRING
schemaVersion: 2.1.0
sdkVersion: kfp-2.4.0

4.Submit pipeline thorugh WebUI
4a Create Experiment
4b Upload Pipeline
4c Run Pipeline for given experiment

(Standard steps, haven’t done print screens but if needed will repeat and generate those)

Pipeline fails with

time="2023-11-12T19:54:25.000Z" level=info msg="capturing logs" argo=true
2
time="2023-11-12T19:54:25.052Z" level=info msg="capturing logs" argo=true
3
I1112 19:54:25.083247 34 launcher_v2.go:90] input ComponentSpec:{
4
"outputDefinitions": {
5
"artifacts": {
6
"html_artifact": {
7
"artifactType": {
8
"schemaTitle": "system.HTML",
9
"schemaVersion": "0.0.1"
10
}
11
}
12
}
13
},
14
"executorLabel": "exec-html-visualization"
15
}
16
I1112 19:54:25.084122 34 cache.go:139] Cannot detect ml-pipeline in the same namespace, default to ml-pipeline.kubeflow:8887 as KFP endpoint.
17
I1112 19:54:25.084154 34 cache.go:116] Connecting to cache endpoint ml-pipeline.kubeflow:8887
18
I1112 19:54:25.237176 34 object_store.go:306] Cannot detect minio-service in the same namespace, default to minio-service.kubeflow:9000 as MinIO endpoint.
19
I1112 19:54:25.273982 34 launcher_v2.go:151] publish success.
20
F1112 19:54:25.274010 34 main.go:49] failed to execute component: unable to create directory "/minio/mlpipeline/v2/artifacts/hello-pipeline/605cef82-4331-4d0b-8f34-888485a3bb77/html-visualization" for output artifact "html_artifact": mkdir /minio: permission denied
21
time="2023-11-12T19:54:26.057Z" level=info msg="sub-process exited" argo=true error="<nil>"
22
Error: exit status 1
23
time="2023-11-12T19:54:27.006Z" level=info msg="sub-process exited" argo=true error="<nil>"
24
Error: exit status 1

25

What did you expect to happen:

Pipeline will succeed and html artifict will be saved to minio. 

Anything else you would like to add:

This is on the back of discussion we had in:
#2442

Hope it helps in kubeflow/kubeflow#6662:

Try out kfp v2 rootless fix(backend): set correct permissions for local directory pipelines#7916

As per conversation we had in #2442, does this mean this issue rely on KFP rootless (#2528) to be delivered or it should work based on this PR:
kubeflow/pipelines#6530 ? 

Environment:

minikube on GCE VM
K8s 1.25
Kubeflow 1.8, manifest deployment - default config

The text was updated successfully, but these errors were encountered:

juliusvonkohout · 2023-11-29T17:00:01Z

/close

Tracked in kubeflow/pipelines#6530

google-oss-prow · 2023-11-29T17:00:05Z

@juliusvonkohout: Closing this issue.

In response to this:

/close

Tracked in kubeflow/pipelines#6530

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jmaunon · 2024-01-14T22:28:25Z

@szymek116 , did you find a work around?
I have installed kubeflow manifests 1.8.0 (KFP 2.0.3) and I do not how to fix even manually.

cc: @juliusvonkohout

juliusvonkohout · 2024-01-15T08:56:43Z

@jmaunon did you see "Tracked in kubeflow/pipelines#6530" ?

This is definitely the wrong repository to track this.

jmaunon · 2024-01-15T09:15:59Z

Thanks @juliusvonkohout . Yes, saw that tracked issue but still do not know how to solve.

I have created a new issue (I mentioned you) to show my code and logs in case it can be helpful.

Thanks

google-oss-prow bot added the kind/bug label Nov 21, 2023

google-oss-prow bot closed this as completed Nov 29, 2023

jmaunon mentioned this issue Jan 15, 2024

[backend] Unable to create directory in Minio when using Artifacts: Permission denied kubeflow/pipelines#10397

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running KFP v2 with non rooted image fails (using artefacts in pipeline) #2573

Running KFP v2 with non rooted image fails (using artefacts in pipeline) #2573

szymek116 commented Nov 21, 2023

juliusvonkohout commented Nov 29, 2023

google-oss-prow bot commented Nov 29, 2023

jmaunon commented Jan 14, 2024

juliusvonkohout commented Jan 15, 2024

jmaunon commented Jan 15, 2024

Running KFP v2 with non rooted image fails (using artefacts in pipeline) #2573

Running KFP v2 with non rooted image fails (using artefacts in pipeline) #2573

Comments

szymek116 commented Nov 21, 2023

juliusvonkohout commented Nov 29, 2023

google-oss-prow bot commented Nov 29, 2023

jmaunon commented Jan 14, 2024

juliusvonkohout commented Jan 15, 2024

jmaunon commented Jan 15, 2024