Implement reference Unstructured store API to upload TaskRun logs GCS store

[tejal29]

Expected Behavior

The Pipeline TaskRun logs should be uploaded to an endpoint and available to download later.
In our initial reference implementation we should support uploading to GCS. In the long run we should support other kinds of stores, and provide a default that does not require GCS.

Actual Behavior

As of #167 the logs will be streamed to a PVC. This volume will continue to exist after the TaskRun has completed. Once this task is done, that PVC should no longer be needed. (This functionality was removed in #443)

Since we moved from init containers to containers for steps in #564 logs are available via the pod logs through kube, however there are still only limited guarantees about how long the logs will be available for.

Steps to Reproduce the Problem

1. Create a Task
2. Create a TaskRun
3. Wait for TaskRun to complete
4. Downloaded the uploaded results.

Additional Info

[bobcatfish]

@tejal29 this is a stretch goal for the milestone so I'm going to remove it from the required milestone tasks

[tejal29]

After feedback with Build working group, we decided to go with following approach.
The API will provide a Sink interface.
A Sink Service Definition will contain

• Endpoint : This is url where the sink is hosted.
• Path to upload to: for now we will be only focusing on uploading logs.
  • Note: We will add support for uploading the Status for TaskRun or PipelineRun in subsequent iteration.
    When we want to upload sat task runs logs, the reconciler will call the service which is running at Endpoint and upload all the logs at path prefix <Path> <Endpoint>:<Path>/<taskruns><task run id>

This will be a Https Service running in our cluster or any other cluster which your cluster can access.

The Sink Base definition will look like this.

package sink
type URL string
type Base struct {
   Endpoint URL
   Path String
}

The reason, we have Path same service can be used to upload logs for multiple projects. They will reside in different Paths and not overlap.

e.g.: I have implemented a GCSSink and installed in my cluster. It is running at "104.198.205.71:8080".
The GCS sink defintion will look this

package sink

type GCS struct {
  *sink.Base
   project string //GCP project.
}

I can define two GCS Sinks which point to 2 buckets "cluster1", "cluster2"

sink1 := sink.GCS {
  Endpoint: "104.198.205.71:8080",
  Path: "cluster1",
  Project: "test1",  // only test1 has write access to gcs bucket gs://cluster1
}

sink2 := sink.GCS {
  Endpoint: "104.198.205.71:8080",
  Path: "cluster2",
 Project: "test2",  // Similarly, only test2 has write access to gcs bucket gs://cluster2
}

Along with that, Sink Interface need to handle 4 url requests "upload/taskruns/", "download/taskruns/" , "upload/pipelineruns/", "download/taskruns/"
(We can also support, Upload and Get with run type. Its up to the implementer )
Note: We can support partial upload and download of logs later.

The Task Reconciler will now make a HTTP request to this sink.Endpoint/sink.Path/upload/taskruns/<id=x>,<contentstream=>

The Design Question over here is how to Define a Sink for a pipeline or a Task run.
Should sink be defined cluster wide Custom Resource?

• this would mean, we will create a new custom resource for Sink.
• Admins can create multiple Sinks.
• The Reconciler will fetch all sinks installed in your cluster and then upload logs to all sinks.

Should Sink be defined per Pipeline or Task?

• this would Sink will added to PipelineParams
• this would mean, now task and task run definition will list the sink it would want to sink the results too

/cc @imjasonh and @bobcatfish and @aaron-prindle does this all make sense?

[aaron-prindle]

Nice! These are some initial thoughts/questions:

1. For something like a GCS sink, how is authorization done for the log uploading? I think GCS w/ GKE will just work but I'm wondering about other clouds/providers?
2. Is there a default path: value that we should populate for users is none is supplied? (path is optional?)
3. For GCS, project should maybe be renamed bucket as I think you can have multiple GCS buckets/project. Also would the endpoint there be the gs:// url or the full url?

[tejal29]

Nice! These are some initial thoughts/questions:

1. For something like a GCS sink, how is authorization done for the log uploading? I think GCS w/ GKE will just work but I'm wondering about other clouds/providers?

Yes you would need like Credentials file added to sink.GCS definition and then pass that along?
Maybe it could be a k8 ConfigMap object and we define the name of the ConfigMap in the GCS sink definition.

2. Is there a default path: value that we should populate for users is none is supplied? (path is optional?)

not sure, what would happen if we provide a default path. We we have taskruns with same id running in separate clusters. They might end up writing to same path. Maybe we could add some validation to make sure path is always specified.

3. For GCS, project should maybe be renamed bucket as I think you can have multiple GCS buckets/project. Also would the endpoint there be the gs:// url or the full url?

Ahh! For GCS i thinking path represents bucket and then project was something i saw in the go storage api.

       ctx := context.Background()

        // For API packages whose import path is starting with "cloud.google.com/go",
        // such as cloud.google.com/go/storage in this case, if there are no credentials
        // provided, the client library will look for credentials in the environment.
        storageClient, err := storage.NewClient(ctx)
        if err != nil {
                log.Fatal(err)
        }

        it := storageClient.Buckets(ctx, "project-id")
        for {
                bucketAttrs, err := it.Next()
                if err == iterator.Done {
                        break
                }
                if err != nil {
                        log.Fatal(err)
                }
                fmt.Println(bucketAttrs.Name)
        }

The endpoint will be actually GCS Sink Implementation Http Service like "10.x.x.x:8080" which will have all the code to upload and download content from GCS.
We are providing a GCS implementation which others can use.
Users will have to write Sink Implementation and deploy it in their cluster as a HTTP service. They have to make sure they handle "upload/taskruns", "download/taskruns" request.
The TaskController is now agnostic to what service implements.

[ghost]

I'm closing this issue out as we have now circulated a design doc for logging in Tekton and the utility of information retained in this issue is limited due to its age.

I've opened #1155 to encompass the work of validating and implementing the proposed design and encourage anyone looking to get involved on this topic to add commentary, use cases and counterpoints to the design doc or github issue linked above. Cheers!