Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ECS] Support Plan Preview for ECS #4881

Merged
merged 40 commits into from
Apr 30, 2024
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
42aa504
Impl ECS's Diff func of taskDef and serviceDef
t-kikuc Apr 8, 2024
52d89e9
Add ECSManifest's cache
t-kikuc Apr 8, 2024
841459b
Impl ECSDiff for planpreview
t-kikuc Apr 8, 2024
8c2157d
Enable ECS PlanPreview
t-kikuc Apr 8, 2024
f99bebe
Draft: diff test
t-kikuc Apr 9, 2024
8190daa
Add and fix tests of diff with TODO of lowerCamelCase
t-kikuc Apr 10, 2024
98b740b
fix tests: add params to servicedef
t-kikuc Apr 10, 2024
43d3ff8
Remove wip comments
t-kikuc Apr 11, 2024
b416577
Skip summarizing if no changes were detected
t-kikuc Apr 12, 2024
5f4f2b9
Add tags to test data
t-kikuc Apr 18, 2024
46cff87
removed an unnecessary test case
t-kikuc Apr 18, 2024
5325c36
Clean comments, removing TODO
t-kikuc Apr 18, 2024
0b33d5a
remove an unnecessary blank line in the test data
t-kikuc Apr 18, 2024
00c9dd8
Modify ApiVersion and Kind to apparent dummy
t-kikuc Apr 18, 2024
c9f2975
Add DiffStructs() for comparing non-k8s manifests
t-kikuc Apr 18, 2024
7613dd5
removed unstructured.Unstructured from ECS
t-kikuc Apr 18, 2024
17b8dfe
Separate DiffByCommand() to diff package
t-kikuc Apr 18, 2024
6b07291
Fix diff render output: split ServiceDef and TaskDef sections
t-kikuc Apr 18, 2024
9ff2fcf
Add a testcase to planpreview_test for ECS
t-kikuc Apr 18, 2024
323b4bb
Merge branch 'master' into ecs-plan-preview
khanhtc1202 Apr 19, 2024
b174d23
Combine unnecessary DiffBytesByCommand() to DiffByCommand()
t-kikuc Apr 23, 2024
e2a38f0
Merge branch 'ecs-plan-preview' of https://github.com/pipe-cd/pipecd …
t-kikuc Apr 23, 2024
dd40abb
Remove IgnorePath from comparing ECS Manifests
t-kikuc Apr 23, 2024
59f3171
rename func to DiffStructureds()
t-kikuc Apr 23, 2024
77dcef7
add comment of which func to use
t-kikuc Apr 23, 2024
9279185
Removed LoadECSManifest() func
t-kikuc Apr 24, 2024
db54c29
Rename test func
t-kikuc Apr 25, 2024
eca1b74
Add tests of diffbycommand
t-kikuc Apr 25, 2024
46233d0
Removed duplicated tests
t-kikuc Apr 25, 2024
97405b4
Rename ECSManifest to ECSManifests
t-kikuc Apr 25, 2024
33fdec1
Merge branch 'master' into ecs-plan-preview
t-kikuc Apr 25, 2024
3fee826
Removed an unnecessary variable
t-kikuc Apr 26, 2024
0cb5d29
Renamed func to 'renderByCommand'
t-kikuc Apr 26, 2024
c9764e4
Add check of existence of the command
t-kikuc Apr 26, 2024
55b89b6
Rename func to 'RenderByCommand'
t-kikuc Apr 26, 2024
c38c07b
Moved func 'RenderByCommand()' to renderer.go
t-kikuc Apr 26, 2024
6f542c6
Merge branch 'ecs-plan-preview' of https://github.com/pipe-cd/pipecd …
t-kikuc Apr 26, 2024
e252b8d
Merge branch 'master' into ecs-plan-preview
t-kikuc Apr 26, 2024
64efe0a
Fix nits: removed unnecessary if-state
t-kikuc Apr 30, 2024
a161cba
Merge branch 'ecs-plan-preview' of https://github.com/pipe-cd/pipecd …
t-kikuc Apr 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion pkg/app/pipectl/cmd/planpreview/planpreview_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,13 @@ NOTE: An error occurred while building plan-preview for applications of the foll
ApplicationKind: model.ApplicationKind_CLOUDRUN,
Error: "missing key",
},
{
ApplicationId: "app-5",
ApplicationName: "app-5",
ApplicationUrl: "https://pipecd.dev/app-5",
ApplicationKind: model.ApplicationKind_ECS,
Error: "wrong application configuration",
},
},
},
{
Expand Down Expand Up @@ -196,14 +203,17 @@ changes-1
changes-2
---DETAILS_END---

NOTE: An error occurred while building plan-preview for the following 2 applications:
NOTE: An error occurred while building plan-preview for the following 3 applications:

1. app: app-3, env: env-3, kind: TERRAFORM
reason: wrong application configuration

2. app: app-4, kind: CLOUDRUN
reason: missing key

3. app: app-5, kind: ECS
reason: wrong application configuration

NOTE: An error occurred while building plan-preview for applications of the following 2 Pipeds:

1. piped: piped-name-1 (piped-1)
Expand Down
2 changes: 2 additions & 0 deletions pkg/app/piped/planpreview/builder.go
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,8 @@
dr, err = b.terraformDiff(ctx, app, targetDSP, &buf)
case model.ApplicationKind_CLOUDRUN:
dr, err = b.cloudrundiff(ctx, app, targetDSP, preCommit, &buf)
case model.ApplicationKind_ECS:
dr, err = b.ecsdiff(ctx, app, targetDSP, preCommit, &buf)

Check warning on line 253 in pkg/app/piped/planpreview/builder.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/planpreview/builder.go#L252-L253

Added lines #L252 - L253 were not covered by tests
default:
// TODO: Calculating planpreview's diff for other application kinds.
dr = &diffResult{
Expand Down
126 changes: 126 additions & 0 deletions pkg/app/piped/planpreview/ecsdiff.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
// Copyright 2024 The PipeCD Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package planpreview

import (
"bytes"
"context"
"fmt"
"io"

"github.com/pipe-cd/pipecd/pkg/app/piped/deploysource"
provider "github.com/pipe-cd/pipecd/pkg/app/piped/platformprovider/ecs"
"github.com/pipe-cd/pipecd/pkg/diff"
"github.com/pipe-cd/pipecd/pkg/model"
)

func (b *builder) ecsdiff(
ctx context.Context,
app *model.Application,
targetDSP deploysource.Provider,
lastCommit string,
buf *bytes.Buffer,
) (*diffResult, error) {
var (
oldManifest, newManifest provider.ECSManifest
err error
)

newManifest, err = b.loadECSManifest(ctx, *app, targetDSP)
if err != nil {
fmt.Fprintf(buf, "failed to load ecs manifest at the head commit (%v)\n", err)
return nil, err
}

Check warning on line 45 in pkg/app/piped/planpreview/ecsdiff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/planpreview/ecsdiff.go#L35-L45

Added lines #L35 - L45 were not covered by tests

if lastCommit == "" {
fmt.Fprintf(buf, "failed to find the commit of the last successful deployment")
return nil, fmt.Errorf("cannot get the old manifest without the last successful deployment")
}

Check warning on line 50 in pkg/app/piped/planpreview/ecsdiff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/planpreview/ecsdiff.go#L47-L50

Added lines #L47 - L50 were not covered by tests

runningDSP := deploysource.NewProvider(
b.workingDir,
deploysource.NewGitSourceCloner(b.gitClient, b.repoCfg, "running", lastCommit),
*app.GitPath,
b.secretDecrypter,
)

oldManifest, err = b.loadECSManifest(ctx, *app, runningDSP)
if err != nil {
fmt.Fprintf(buf, "failed to load ecs manifest at the running commit (%v)\n", err)
return nil, err
}

Check warning on line 63 in pkg/app/piped/planpreview/ecsdiff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/planpreview/ecsdiff.go#L52-L63

Added lines #L52 - L63 were not covered by tests

result, err := provider.Diff(
oldManifest,
newManifest,
diff.WithEquateEmpty(),
diff.WithCompareNumberAndNumericString(),
)
if err != nil {
fmt.Fprintf(buf, "failed to compare manifests (%v)\n", err)
return nil, err
}

Check warning on line 74 in pkg/app/piped/planpreview/ecsdiff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/planpreview/ecsdiff.go#L65-L74

Added lines #L65 - L74 were not covered by tests

if result.NoChange() {
fmt.Fprintln(buf, "No changes were detected")
return &diffResult{
summary: "No changes were detected",
noChange: true,
}, nil
}

Check warning on line 82 in pkg/app/piped/planpreview/ecsdiff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/planpreview/ecsdiff.go#L76-L82

Added lines #L76 - L82 were not covered by tests

summary := fmt.Sprintf("%d changes were detected", len(result.Diff.Nodes()))
details := result.Render(provider.DiffRenderOptions{
UseDiffCommand: true,
})
fmt.Fprintf(buf, "--- Last Deploy\n+++ Head Commit\n\n%s\n", details)

return &diffResult{
summary: summary,
}, nil

Check warning on line 92 in pkg/app/piped/planpreview/ecsdiff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/planpreview/ecsdiff.go#L84-L92

Added lines #L84 - L92 were not covered by tests
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] maybe be able to reduce the variable

Suggested change
summary := fmt.Sprintf("%d changes were detected", len(result.Diff.Nodes()))
details := result.Render(provider.DiffRenderOptions{
UseDiffCommand: true,
})
fmt.Fprintf(buf, "--- Last Deploy\n+++ Head Commit\n\n%s\n", details)
return &diffResult{
summary: summary,
}, nil
details := result.Render(provider.DiffRenderOptions{
UseDiffCommand: true,
})
fmt.Fprintf(buf, "--- Last Deploy\n+++ Head Commit\n\n%s\n", details)
return &diffResult{
summary: fmt.Sprintf("%d changes were detected", len(result.Diff.Nodes())),
}, nil

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I fixed

3fee826


}

func (b *builder) loadECSManifest(ctx context.Context, app model.Application, dsp deploysource.Provider) (provider.ECSManifest, error) {
commit := dsp.Revision()
cache := provider.ECSManifestCache{
AppID: app.Id,
Cache: b.appManifestsCache,
Logger: b.logger,
}

manifest, ok := cache.Get(commit)
if ok {
return manifest, nil
}

Check warning on line 107 in pkg/app/piped/planpreview/ecsdiff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/planpreview/ecsdiff.go#L96-L107

Added lines #L96 - L107 were not covered by tests

ds, err := dsp.Get(ctx, io.Discard)
if err != nil {
return provider.ECSManifest{}, err
}

Check warning on line 112 in pkg/app/piped/planpreview/ecsdiff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/planpreview/ecsdiff.go#L109-L112

Added lines #L109 - L112 were not covered by tests

appCfg := ds.ApplicationConfig.ECSApplicationSpec
if appCfg == nil {
return provider.ECSManifest{}, fmt.Errorf("malformed application configuration file")
}

Check warning on line 117 in pkg/app/piped/planpreview/ecsdiff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/planpreview/ecsdiff.go#L114-L117

Added lines #L114 - L117 were not covered by tests

manifest, err = provider.LoadECSManifest(ds.AppDir, appCfg.Input.TaskDefinitionFile, appCfg.Input.ServiceDefinitionFile)
if err != nil {
return provider.ECSManifest{}, err
}

Check warning on line 122 in pkg/app/piped/planpreview/ecsdiff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/planpreview/ecsdiff.go#L119-L122

Added lines #L119 - L122 were not covered by tests

cache.Put(commit, manifest)
return manifest, nil

Check warning on line 125 in pkg/app/piped/planpreview/ecsdiff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/planpreview/ecsdiff.go#L124-L125

Added lines #L124 - L125 were not covered by tests
}
68 changes: 68 additions & 0 deletions pkg/app/piped/platformprovider/ecs/cache.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
// Copyright 2024 The PipeCD Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package ecs

import (
"errors"
"fmt"

"go.uber.org/zap"

"github.com/pipe-cd/pipecd/pkg/cache"
)

type ECSManifestCache struct {
AppID string
Cache cache.Cache
Logger *zap.Logger
}

func (c ECSManifestCache) Get(commit string) (ECSManifest, bool) {
key := ecsManifestCacheKey(c.AppID, commit)
item, err := c.Cache.Get(key)
if err == nil {
return item.(ECSManifest), true
}

Check warning on line 37 in pkg/app/piped/platformprovider/ecs/cache.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/platformprovider/ecs/cache.go#L32-L37

Added lines #L32 - L37 were not covered by tests

if errors.Is(err, cache.ErrNotFound) {
c.Logger.Info("ecs manifest were not found in cache",
zap.String("app-id", c.AppID),
zap.String("commit-hash", commit),
)
return ECSManifest{}, false
}

Check warning on line 45 in pkg/app/piped/platformprovider/ecs/cache.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/platformprovider/ecs/cache.go#L39-L45

Added lines #L39 - L45 were not covered by tests

c.Logger.Error("failed while retrieving ecs manifest from cache",
zap.String("app-id", c.AppID),
zap.String("commit-hash", commit),
zap.Error(err),
)
return ECSManifest{}, false

Check warning on line 52 in pkg/app/piped/platformprovider/ecs/cache.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/platformprovider/ecs/cache.go#L47-L52

Added lines #L47 - L52 were not covered by tests
}

func (c ECSManifestCache) Put(commit string, sm ECSManifest) {
key := ecsManifestCacheKey(c.AppID, commit)
if err := c.Cache.Put(key, sm); err != nil {
c.Logger.Error("failed while putting ecs manifest from cache",
zap.String("app-id", c.AppID),
zap.String("commit-hash", commit),
zap.Error(err),
)
}

Check warning on line 63 in pkg/app/piped/platformprovider/ecs/cache.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/platformprovider/ecs/cache.go#L55-L63

Added lines #L55 - L63 were not covered by tests
}

func ecsManifestCacheKey(appID, commit string) string {
return fmt.Sprintf("%s/%s", appID, commit)

Check warning on line 67 in pkg/app/piped/platformprovider/ecs/cache.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/platformprovider/ecs/cache.go#L66-L67

Added lines #L66 - L67 were not covered by tests
}
105 changes: 105 additions & 0 deletions pkg/app/piped/platformprovider/ecs/diff.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
// Copyright 2024 The PipeCD Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package ecs

import (
"bytes"
"fmt"
"strings"

"github.com/pipe-cd/pipecd/pkg/diff"
)

const (
diffCommand = "diff"
)

type DiffResult struct {
Diff *diff.Result
Old ECSManifest
New ECSManifest
}

func (d *DiffResult) NoChange() bool {
return len(d.Diff.Nodes()) == 0
}

func Diff(old, new ECSManifest, opts ...diff.Option) (*DiffResult, error) {
key := old.ServiceDefinition.ServiceName
if key == nil {
return nil, fmt.Errorf("service name is required in the old service definition")
}

Check warning on line 43 in pkg/app/piped/platformprovider/ecs/diff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/platformprovider/ecs/diff.go#L42-L43

Added lines #L42 - L43 were not covered by tests

d, err := diff.DiffStructs(old, new, *key, opts...)
if err != nil {
return nil, err
}

Check warning on line 48 in pkg/app/piped/platformprovider/ecs/diff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/platformprovider/ecs/diff.go#L47-L48

Added lines #L47 - L48 were not covered by tests

if !d.HasDiff() {
return &DiffResult{Diff: d}, nil
}

ret := &DiffResult{
Old: old,
New: new,
Diff: d,
}
return ret, nil
}

type DiffRenderOptions struct {
// If true, use "diff" command to render.
UseDiffCommand bool
}

func (d *DiffResult) Render(opt DiffRenderOptions) string {
var b strings.Builder
opts := []diff.RenderOption{
diff.WithLeftPadding(1),
}
renderer := diff.NewRenderer(opts...)
if !opt.UseDiffCommand {
b.WriteString(renderer.Render(d.Diff.Nodes()))
} else {
d, err := diffByCommand(diffCommand, d.Old, d.New)
if err != nil {
b.WriteString(fmt.Sprintf("An error occurred while rendering diff (%v)", err))

Check warning on line 78 in pkg/app/piped/platformprovider/ecs/diff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/platformprovider/ecs/diff.go#L78

Added line #L78 was not covered by tests
} else {
b.Write(d)
}
}
b.WriteString("\n")

return b.String()
}

func diffByCommand(command string, old, new ECSManifest) ([]byte, error) {
taskDiff, err := diff.DiffByCommand(command, old.TaskDefinition, new.TaskDefinition)
if err != nil {
return nil, err
}

serviceDiff, err := diff.DiffByCommand(command, old.ServiceDefinition, new.ServiceDefinition)
if err != nil {
return nil, err
}

Check warning on line 97 in pkg/app/piped/platformprovider/ecs/diff.go

View check run for this annotation

Codecov / codecov/patch

pkg/app/piped/platformprovider/ecs/diff.go#L96-L97

Added lines #L96 - L97 were not covered by tests

return bytes.Join([][]byte{
[]byte("# 1. ServiceDefinition"),
serviceDiff,
[]byte("\n# 2. TaskDefinition"),
taskDiff,
}, []byte("\n")), nil
}
Loading
Loading